Method and system for determining poses for camera calibration

ABSTRACT

A method and system for determining poses for camera calibration is presented. The system determines a range of pattern orientations for performing the camera calibration, and determines a surface region on a surface of an imaginary sphere, which represents possible pattern orientations for the calibration pattern. The system determines a plurality of poses for the calibration pattern to adopt. The plurality of poses may be defined by respective combinations of a plurality of respective locations within the camera field of view and a plurality of respective sets of pose angle values. Each set of pose angle values of the plurality of respective sets may be based on a respective surface point selected from within the surface region on the surface of the imaginary sphere. The system outputs a plurality of robot movement commands based on the plurality of poses that are determined.

FIELD OF THE INVENTION

The present invention is directed to a method and system for determiningposes for camera calibration and for robot control.

BACKGROUND

As automation becomes more common, robots are being used in moreenvironments, such as in warehousing and manufacturing environments. Forinstance, robots may be used to load items onto or off of a pallet in awarehouse, or to pick up objects from a conveyor belt in a factory. Themovement of the robot may be fixed, or may be based on an input, such asan image taken by a camera in the warehouse or factory. In the lattersituation, calibration may be performed so as to determine a property ofthe camera, and to determine a relationship between the camera and anenvironment in which the robot is located. The calibration may bereferred to as camera calibration, and may generate calibrationinformation that is used to control the robot based on images capturedby the camera. In some implementations, the camera calibration mayinvolve manual operation by a person, who may manually control movementof the robot, or manually control the camera to capture an image of therobot.

SUMMARY

One aspect of the embodiments herein relates to a computing system or amethod performed by the computing system (e.g., via instructions on anon-transitory computer-readable medium). The computing system maycomprise a communication interface configured to communicate with arobot and with a camera having a camera field of view, wherein the robothas a calibration pattern disposed thereon. The computing system mayfurther have a control circuit configured, when the computing system isin communication with the robot and with the camera, to perform cameracalibration by: determining a range of pattern orientations forperforming the camera calibration, wherein the range of patternorientations is a range of orientations for the calibration pattern;determining a surface region on a surface of an imaginary sphere,wherein the surface of the imaginary sphere represents possible patternorientations for the calibration pattern, and the surface regionrepresents the range of pattern orientations for performing the cameracalibration; determining a plurality of poses for the calibrationpattern to adopt when the camera calibration is being performed, whereinthe plurality of poses are defined by respective combinations of aplurality of respective locations within the camera field of view and aplurality of respective sets of pose angle values, wherein each set ofpose angle values of the plurality of respective sets is based on arespective surface point selected from within the surface region on thesurface of the imaginary sphere; outputting a plurality of robotmovement commands for controlling placement of the calibration pattern,wherein the plurality of robot movement commands are generated based onthe plurality of poses that are determined; receiving a plurality ofcalibration images, wherein each calibration image of the plurality ofcalibration images represents the calibration pattern and is generatedwhile the calibration pattern has a respective pose of the plurality ofposes; and determining an estimate of a camera calibration parameterbased on the plurality of calibration images. The control circuit isfurther configured, after the camera calibration is performed, toreceive a subsequent image from the camera via the communicationinterface, and to output a subsequent robot movement command that isgenerated based on the subsequent image and based on the estimate of thecamera calibration parameter.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features, objects and advantages of theinvention will be apparent from the following description of embodimentshereof as illustrated in the accompanying drawings. The accompanyingdrawings, which are incorporated herein and form a part of thespecification, further serve to explain the principles of the inventionand to enable a person skilled in the pertinent art to make and use theinvention. The drawings are not to scale.

FIG. 1 depicts a block diagram of systems in which camera calibration isperformed, according to an embodiment herein.

FIG. 2 depicts a block diagram of a computing system configured todetermine poses for performing camera calibration, according to anembodiment herein.

FIGS. 3A and 3B depict a system in which camera calibration isperformed, and in which a calibration pattern can adopt various poses,according to an embodiment herein.

FIGS. 4A illustrates an angle value of a pose angle between acalibration pattern and a camera, according to an embodiment herein.

FIGS. 4B and 4C depict a normal vector of the calibration pattern, acamera coordinate system, and a pattern coordinate system, according toan embodiment herein.

FIGS. 5A-5C depict various pose angles for a calibration pattern,according to an embodiment herein.

FIGS. 6A-6C depict various pose angles for a calibration pattern,according to an embodiment herein.

FIGS. 7A-7C illustrate an imaginary sphere that represent possiblepattern orientations of a calibration pattern, and surface points on theimaginary sphere.

FIG. 8A depicts an example of constraints on pose angle values,according to an embodiment hereof.

FIGS. 8B and 8C illustrate an imaginary sphere that represent possibleorientations of a calibration pattern, a surface region that representsa range of pattern orientations for performing camera calibration, andsurface points on the imaginary sphere, according to an embodimenthereof.

FIG. 9 depicts a flow diagram of an example method for determining posesfor performing camera calibration, according to an embodiment hereof.

FIGS. 10A and 10B depict surface that points that are uniformlydistributed within a surface region of an imaginary sphere thatrepresents pattern orientations, according to an embodiment hereof.

FIGS. 11A and 11B depict an example grid that divides a space in acamera field of view, according to embodiments hereof.

FIGS. 12A-12C illustrate examples of a Latin square spatial distributionfor various poses, according to embodiments hereof.

FIGS. 13A and 13B illustrate examples of a stratified spatialdistribution for various poses, according to embodiments hereof.

FIGS. 14A and 14B illustrate examples of a random spatial distributionfor various poses, according to an embodiment hereof.

FIGS. 15A-15C illustrate various spatial distributions forrobot-achievable candidate poses, according to embodiments hereof.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and isnot intended to limit the invention or the application and uses of theinvention. Furthermore, there is no intention to be bound by anyexpressed or implied theory presented in the preceding technical field,background, brief summary or the following detailed description.

Embodiments described herein relate to determining poses for performingcamera calibration. A pose may refer to, e.g., an orientation (which maybe referred to as pattern orientation) at which a calibration pattern isplaced, a location at which the calibration pattern is placed, or acombination thereof. A camera may photograph or otherwise image thecalibration pattern while the calibration pattern has that pose, so asto generate a calibration image corresponding to the pose, and thecalibration image may be used to perform the camera calibration.Performing the camera calibration may, e.g., involve estimating aproperty of the camera, and/or a relationship between the camera and itsenvironment. After the camera calibration is performed, images generatedby the camera may facilitate control of a robot that is used to interactwith objects in the environment of the camera. For instance, the robotmay be used to pick up a package in a warehouse, wherein movement of anarm or other component of the robot may be based on images of thepackage generated by the camera.

One aspect of the embodiments herein relates to attempting to achieve adistribution of poses in which the poses are generally spread out interms of location of the calibration pattern and/or pattern orientation.If the poses instead have a distribution in which the poses areconcentrated in certain regions, or are concentrated around certainpattern orientations in a desired range of pattern orientations, theresulting calibration images may capture certain camera behavior thatare manifested when photographed objects are at those regions and/ororientations, but may miss camera behavior corresponding to photographedobjects being at other regions and/or orientations. Determining theposes in a manner that instead spreads out the poses, so as to create,e.g., a more uniform distribution of the poses in terms of locationand/or orientation, may cause the resulting calibration images to morecompletely or more accurately capture camera behavior. For instance, ifthe camera behavior is lens distortion that can be introduced by a lensof the camera, spreading out the poses of the calibration pattern mayallow the calibration pattern to have diverse poses, and to bephotographed or otherwise imaged at diverse locations and/or patternorientations. Such diverse poses may render the resulting calibrationimages more likely to capture a greater number of ways in which the lensdistortion is manifested. Such calibration images may allow the lensdistortion, or another property of the camera, to be characterized orotherwise estimated in a more complete and accurate manner.

One aspect of the embodiments herein relate to determining pose anglevalues for different poses of the calibration pattern, and morespecifically to doing so in a manner that achieves a desireddistribution for pattern orientations of the calibration pattern. Thedistribution of pattern orientations may refer to a distribution ofdirections in which the calibration pattern is oriented. For instance, agenerally uniform distribution within a desired range of patternorientations may refer to a distribution in which the calibrationpattern has directions that are within a desired range of directions,and are generally evenly distributed among the desired range ofdirections, wherein the desired range of pattern orientations may bedefined by the desired range of directions.

In an embodiment, the pose angle value discussed above may be an anglevalue of a pose angle, which may be an angle between the calibrationpattern and a frame of reference, such as an optical axis of the camera.The pose angle may be used to control tilting of the calibration patternrelative to, e.g., the camera (such a tilt may be referred to as arelative tilt). In an embodiment, multiple pose angles may be used tocontrol tilting of the calibration pattern, and a set of respective poseangle values for the multiple pose angles may be used to control adirection and amount of the relative tilt of the calibration pattern. Insome cases, a set of pattern orientations may be determined bydetermining a set of respective pose angle values for each of themultiple pose angles individually, according to a desired distribution(e.g., a uniform distribution). However, such an approach may notactually achieve the desired distribution for the set of patternorientations. For instance, if a pattern orientation is controlled bythree pose angles, determining a set of pose angle values for each ofthe three pose angles individually, according to a uniform distribution,may not actually lead to a uniform distribution for the resulting set ofpattern orientations. Thus, one aspect of the embodiments herein relateto determining a pose angle value for a pose angle by initiallydetermining a pattern orientation that is consistent with a desireddistribution, and then determining the pose angle value based on thedesired distribution.

In an embodiment, determining a pattern orientation that is consistentwith a desired distribution may involve selecting a surface point thatis on an imaginary sphere. The surface point may be a point on a surfaceof the imaginary sphere, which may represent possible patternorientations for a calibration pattern, and more specifically mayrepresent directions at which a normal vector of the calibration patterncan point. In some cases, a center of the imaginary sphere may be at oneendpoint of the normal vector, and the imaginary sphere may have asurface that is a loci of points that can be pointed at or moregenerally directed towards by the other endpoint of the normal vector.In some cases, a region on the surface of the imaginary sphere (whichmay be referred to as a surface region) may represent a desired range ofpattern orientations, and surface points within the surface region mayrepresent respective pattern orientations within the desired range. Inan embodiment, the surface point may be selected from the surface of theimaginary sphere, and more specifically from within the surface region,according to a desired distribution. For example, selecting the surfacepoint according to a desired uniform distribution may involve samplingsurface points within the surface region to select one of those surfacepoints, wherein the sampling may be done in a manner such that each ofthe surface points within the surface region is equally likely to beselected. In this embodiment, a pose angle value for a pose angle may bedetermined based on the selected surface point. If the patternorientation is controlled by multiple pose angles, then a respectivepose angle value may be determined for each of the multiple pose anglesbased on the selected surface point. If a plurality of patternorientations are determined in the above manner for a plurality ofrespective poses, the plurality of pattern orientations may more likelyhave a desired distribution, such as a uniform distribution.

One aspect of the embodiments herein relates to determining respectivelocations for a plurality of poses in a manner such that the pluralityof poses are spread out within the camera's field of view (also referredto as a camera field of view). Each of the determined locations may, insome cases, be combined with a respective set of pose angle values toform a pose for the calibration pattern. The respective set of poseangle values may be determined using, e.g., the manner described above.In an embodiment, a space within the camera's field of view may bedivided into a grid that has one or more layers and has multiple rowsand multiple columns. In some cases, determining the respectivelocations may involve attempting to find locations that will achieve afirst spatial distribution which will place the plurality of poses atdiverse regions. If the first spatial distribution cannot be achieved,the determination may further involve attempting to find locations toachieve a second spatial distribution that may also attempt to place theplurality of poses at diverse regions, but may have less conditions or amore relaxed condition relative to the first spatial distribution. Insome cases, if the first spatial distribution, the second spatialdistribution, and/or another spatial distribution cannot be achieved,the locations for the plurality of poses may be determined to achieve arandom spatial distribution.

In an embodiment, the first spatial distribution may be a distributionin which i) each row in a particular layer of the grid includes only onepose, or includes no more than one pose, and in which ii) each column inthe layer includes only one pose, or includes no more than one pose. Inan embodiment, the second spatial distribution may be a distribution inonly one of the above criteria for the first spatial distribution haveto be satisfied. More specifically, the second spatial distribution maybe a distribution in which i) each row in a particular layer of the gridincludes only one pose, or includes no more than one pose, or ii) eachcolumn in a particular layer includes only one pose, or no more than onepose.

FIG. 1 illustrates a block diagram of a robot operation system 100 forperforming automatic camera calibration. The robot operation system 100includes a robot 150, a computing system 110, and a camera 170. In somecases, the computing system 110 may be configured to control the robot150, and may be referred to in those cases as a robot control system ora robot controller. In an embodiment, the robot operation system 100 maybe located within a warehouse, a manufacturing plant, or other premises.The computing system 110 may be configured to perform cameracalibration, such as by determining calibration information that islater used to control the robot 150. In some cases, the computing system110 is configured both to perform the camera calibration and to controlthe robot 150 based on the calibration information. In some cases, thecomputing system 110 may be dedicated to performing the cameracalibration, and may communicate the calibration information to anothercomputing system that is dedicated to controlling the robot. The robot150 may be positioned based on images captured by the camera 170 and onthe calibration information. In some cases, the computing system 110 maybe part of a vision system that acquires images of an environment inwhich the camera 170 is located.

In an embodiment, the computing system 110 may be configured tocommunicate via a wired or wireless communication with the robot 150 andthe camera 170. For instance, the computing system 110 may be configuredto communicate with the robot 150 and/or the camera 170 via a RS-232interface, a universal serial bus (USB) interface, an Ethernetinterface, a Bluetooth® interface, an IEEE 802.11 interface, or anycombination thereof. In an embodiment, the computing system 110 may beconfigured to communicate with the robot 150 and/or the camera 170 via alocal computer bus, such as a peripheral component interconnect (PCI)bus.

In an embodiment, the computing system 110 may be separate from therobot 150, and may communicate with the robot 150 via the wireless orwired connection discussed above. For instance, the computing system 110may be a standalone computer that is configured to communicate with therobot 150 and the camera 170 via a wired connection or wirelessconnection. In an embodiment, the computing system 110 may be anintegral component of the robot 150, and may communicate with othercomponents of the robot 150 via the local computer bus discussed above.In some cases, the computing system 110 may be a dedicated controlsystem (also referred to as a dedicated controller) that controls onlythe robot 150. In other cases, the computing system 110 may beconfigured to control multiple robots, including the robot 150. In anembodiment, the computing system 110, the robot 150, and the camera 170are located at the same premises (e.g., warehouse). In an embodiment,the computing system 110 may be remote from the robot 150 and the camera170, and may be configured to communicate with the robot 150 and thecamera 170 via a network connection (e.g., local area network (LAN)connection).

In an embodiment, the computing system 110 may be configured to accessand to process calibration images, which are images of a calibrationpattern 160 that is disposed on the robot 150. The computing system 110may access the calibration images by retrieving or, more generallyreceiving, the calibration images from the camera 170 or from anothersource, such as from a storage device or other non-transitorycomputer-readable medium on which the calibration images are stored. Insome instances, the computing system 110 may be configured to controlthe camera 170 to capture such images. For example, the computing system110 may be configured to generate a camera command that causes thecamera 170 to generate an image that captures a scene in a field of viewof the camera 170 (also referred to as a camera field of view), and tocommunicate the camera command to the camera 170 via the wired orwireless connection. The same command may cause the camera 170 to alsocommunicate the image (as image data) back to the computing system 110,or more generally to a storage device accessible by the computing system110. Alternatively, the computing system 110 may generate another cameracommand that causes the camera 170, upon receiving the camera command,to communicate an image(s) it has captured to the computing system 110.In an embodiment, the camera 170 may automatically capture an image of ascene in its camera field of view, either periodically or in response toa defined triggering condition, without needing a camera command fromthe computing system 110. In such an embodiment, the camera 170 may alsobe configured to automatically, without a camera command from thecomputing system 110, communicate the image to the computing system 110or, more generally, to a storage device accessible by the computingsystem 110.

In an embodiment, the computing system 110 may be configured to controlmovement of the robot 150 via movement commands that are generated bythe computing system 110 and communicated over the wired or wirelessconnection to the robot 150. The movement commands may cause the robotto move a calibration pattern 160 disposed on the robot. The calibrationpattern 160 may be permanently disposed on the robot 150, or may be aseparate component that can be attached to and detached from the robot150.

In an embodiment, the camera 170 may be configured to generate orotherwise acquire an image that captures a scene in a camera field ofview, such as by photographing the scene. The image may be formed byimage data, such as an array of pixels. The camera 170 may be a colorimage camera, a grayscale image camera, a depth-sensing camera (e.g., atime-of-flight (TOF) or structured light camera), or any other camera.In an embodiment, the camera 170 may include one or more lenses, animage sensor, and/or any other component. The image sensor may include,e.g., a charge-coupled device (CCD) sensor, a complementary metal oxidesemiconductor (CMOS) sensor, a quanta image sensor (QIS), or any otherimage sensor.

FIG. 2 depicts a block diagram of the computing system 110. Asillustrated in the block diagram, the computing system 110 can include acontrol circuit 111, a communication interface 113, and a non-transitorycomputer-readable medium 115 (e.g., memory). In an embodiment, thecontrol circuit 111 may include one or more processors, a programmablelogic circuit (PLC) or a programmable logic array (PLA), a fieldprogrammable gate array (FPGA), an application specific integratedcircuit (ASIC), or any other control circuit.

In an embodiment, the communication interface 113 may include one ormore components that are configured to communicate with the camera 170and the robot 150. For instance, the communication interface 113 mayinclude a communication circuit configured to perform communication overa wired or wireless protocol. As an example, the communication circuitmay include a RS-232 port controller, a USB controller, an Ethernetcontroller, an IEEE 802.11 controller, a Bluetooth® controller, a PCIbus controller, any other communication circuit, or a combinationthereof.

In an embodiment, the non-transitory computer-readable medium 115 mayinclude an information storage device, such as computer memory. Thecomputer memory may comprise, e.g., dynamic random access memory (DRAM),solid state integrated memory, and/or a hard disk drive (HDD). In somecases, the camera calibration may be implemented throughcomputer-executable instructions (e.g., computer code) stored on thenon-transitory computer-readable medium 115. In such cases, the controlcircuit 111 may include one or more processors configured to execute thecomputer-executable instructions to perform the camera calibration(e.g., the steps illustrated in FIG. 9). In an embodiment, thenon-transitory computer-readable medium may be configured to store oneor more calibration images that are generated by the camera 170.

As stated above, one aspect of the embodiments herein relate todetermining a plurality of poses for the calibration pattern 160. Eachpose may refer to a combination of a location and a pattern orientationof the calibration pattern 160. The plurality of poses may be determinedso as to place the calibration pattern 160 (via the robot 150) atdifferent locations within the camera field of view, and to tilt orotherwise move the calibration pattern 160 to have different patternorientations relative to the camera 170. For instance, FIGS. 3A and 3Bdepict two different poses for a calibration pattern 260 within a robotoperation system 200. The robot operation system 200 may be anembodiment of the robot operation system 100, and includes the computingsystem 110, a camera 270 (which may be an embodiment of the camera 170),a robot 250 (which may be an embodiment of the robot 150), and thecalibration pattern 260, which may be an embodiment of the calibrationpattern 260.

In the example of FIGS. 3A and 3B, the robot 250 may have a robot armthat comprises a plurality of links 254A-254E that are connected byjoints. The robot arm may be configured to move the calibration pattern260. The movement may include placing the calibration pattern 260 atdifferent locations within a camera field of view 272 of the camera 270,and/or tilting the calibration pattern 260 to different orientationsrelative to the camera 270. In some cases, the robot arm may beconfigured to move via rotation of one or more links of the plurality oflinks 254A-254E about one or more of the joints connecting the pluralityof links 254A-254E. In some instances, the robot arm may move inresponse to a movement command (also referred to as a robot movementcommand). For example, the robot 250 may include one or more motors (notshown) that are configured to output rotation at the plurality of jointsconnecting the links 254A-254E, so as to cause at least some of thelinks 254A-254E to rotate. In such an example, the movement command mayinclude one or more motor commands that cause one or more of the motorsto be activated. In some cases, the movement command may be generated bythe computing system 110 and outputted by the computing system 110 tothe robot 150. In some cases, the robot movement command may begenerated by another computing system, and/or by the robot 250.

In an embodiment, the calibration pattern 260, which is an embodiment ofthe calibration pattern 160, may be moved by the robot arm of the robot250 to different poses. More specifically, FIGS. 3A and 3B depict thecalibration pattern 260 having a first pose and a second pose,respectively. The first pose and the second pose may be considereddifferent poses because they have different respective combinations of alocation for the calibration pattern 260 and a pattern orientation ofthe calibration pattern 260.

In an embodiment, the camera 270 may generate or otherwise acquire afirst calibration image that captures the calibration pattern 260 whilethe calibration pattern 260 has the first pose depicted in FIG. 3A, andmay generate or otherwise acquire a second calibration image thatcaptures the calibration pattern 260 while the calibration pattern 260has the second pose depicted in FIG. 3B. Because the first calibrationimage and the second calibration image capture the calibration patternwhile the calibration pattern 260 is at the first pose and the secondpose, respectively, the two poses may also be referred to asimage-captured poses. The first calibration image and the secondcalibration image may be accessed by the computing system 110 or anyother computing system to perform camera calibration.

In an embodiment, the camera calibration may determine, for instance, anestimate of one or more intrinsic camera parameters for the camera 270,and/or a relationship between the camera 270 and its environment. Theone or more intrinsic camera parameters may include, e.g., a projectionmatrix of the camera 270, one or more distortion parameters of thecamera 270, or any combination thereof. The relationship between thecamera 270 and its environment may include, e.g., a matrix thatdescribes a spatial relationship between the camera 270 and the robot250. More specifically, the matrix may describe a spatial relationshipbetween the camera 270 and a world point 294 (which is depicted in FIG.3B), which may be a point that is stationary relative to the base 252 ofthe robot 250. The camera calibration information may subsequently beused to facilitate interaction between the robot 250 and an object, suchas a package in a warehouse. For instance, the camera 270 may beconfigured to generate or otherwise acquire an image of the object, andthe computing system 110 or some other computing system may beconfigured to use the image of the object and the camera calibrationinformation to determine a spatial relationship between the robot 250and the object. Determining the estimate of camera calibrationparameters is discussed in more detail in U.S. patent application Ser.No. 16/295,940, entitled “METHOD AND SYSTEM FOR PERFORMING AUTOMATICCAMERA CALIBRATION FOR ROBOT CONTROL,” which is incorporated byreference herein in its entirety.

In an embodiment, as also stated above, a pattern orientation of thecalibration pattern 160/260 may be controlled by one or more poseangles. Generally speaking, a pose angle may be an angle between thecalibration pattern 160/260 and a reference axis. For instance, FIG. 4Adepicts a pose angle a that is formed between a normal vector 261 of thecalibration pattern 260 and a

_(Reference) axis. The normal vector 261 may be a vector that isorthogonal to a plane defined by the calibration pattern 260 of FIG. 4A.In some instances, the normal vector may be a vector that is coincident,or more generally aligned with, a

_(Pattern) axis, which is discussed below in more detail. The

_(Reference) axis may be a reference axis against which the pose angle αis measured. In the example of FIG. 4A,

_(Reference) may parallel with and/or coincident with a Z-axis for acoordinate system of the camera 270 (also referred to as a cameracoordinate system), wherein the Z-axis is labeled

_(Camera). The camera coordinate system, along with a world coordinatesystem, are illustrated in FIG. 4B. In some cases, if the camera 270 hasone or more lenses,

_(Camera) for the camera coordinate system may be an optical axis of theone or more lenses. Further, if the camera 270 has an image sensor(e.g., a CCD sensor), the X-axis, or

_(Camera), of the camera coordinate system and the Y-axis, or

_(Camera), of the camera coordinate system may define a two-dimensional(2D) image plane of the image sensor. The camera coordinate system mayhave an origin that is located at the one or more lenses, on a surfaceof the image sensor, or at any other location. FIG. 4B furtherillustrates a world coordinate system, which may be a coordinate systemthat has an origin about the world point 294. As depicted in FIG. 4B,the world coordinate system may be defined by the axes

_(World),

_(World),

_(World).

As stated above, the normal vector 261 of the calibration pattern 260may in some cases be coincident with, or more generally parallel with, aZ-axis of a pattern coordinate system, or

_(Pattern). FIG. 4C depicts the coordinate axes of the patterncoordinate system, namely

_(Pattern),

_(Pattern), and

_(Pattern). In some implementations, the calibration pattern 260 mayhave a plurality of pattern elements (e.g., dots) that have knownlocations, or more generally defined locations, in the patterncoordinate system. For instance, the plurality of pattern elements maybe a grid of dots that have predefined spacing. In such implementations,an origin of the pattern coordinate system may be located at one of thepattern elements, or may be located elsewhere. Calibration patterns arediscussed in more detail in U.S. patent application Ser. No. 16/295,940,entitled “METHOD AND SYSTEM FOR PERFORMING AUTOMATIC CAMERA CALIBRATIONFOR ROBOT CONTROL,” the content of which is incorporated by referenceherein in its entirety.

In an embodiment, a pattern orientation of the calibration pattern160/260 may be controlled by a set of pose angles, and more specificallyby a set of respective pose angle values for the pose angles. Forinstance, FIGS. 5A-5C depict using some or all of three pose angles α,β, θ to define a pattern orientation of the calibration pattern 260. Inthis example, pose angle α may be an angle that is formed between thenormal vector 261 and

_(Reference) along a Y-Z plane formed by

_(Reference) and

_(Reference). In this example, the normal vector 261 may be coincidentwith

_(Pattern), such that the pose angle α may also be an angle between

_(Pattern) and

_(Reference). In some instances, the reference axis

_(Reference) may be an axis that is coincident, or more generallyparallel, with the

_(Camera) of FIG. 4B. In some instances, the

_(Reference) axis may be an optical axis of the camera 270. In someinstances,

_(Reference) may be an axis that is parallel with the

_(Camera) axis of FIG. 4B, while

_(Reference) axis may be an axis that is parallel with the

_(Camera) axis of FIG. 4B. In the example depicted in FIG. 5A, the poseangle α may also be defined as an angle between

_(Pattern) and

_(Reference), and/or as an angle that is formed by rotating thecalibration board 260 about the

_(Reference) axis, which may be an axis of rotation for the angle α.

In an embodiment, the pose angle β, which is illustrated in FIG. 5B, maybe an angle that is formed between the normal vector 261 and

_(Reference) along a X-Z plane formed by

_(Reference) and

_(Reference). Because the normal vector 261 may be coincident with

_(Pattern), the pose angle β may also be an angle between

_(Pattern) and

_(Reference). In the example depicted in FIG. 5B, the pose angle β mayalso be defined as an angle between

_(Pattern) and

_(Reference), and/or as an angle that is formed by rotating thecalibration board 260 about the

_(Reference) axis, which may be an axis of rotation for the angle β.

In an embodiment, the pose angle θ, which is illustrated in FIG. 5C, maybe an angle that is formed between

_(Pattern) and

_(Reference), or between

_(Pattern) and

_(Reference), along a X-Y plane formed by

_(Reference) and

_(Reference). In the example depicted in FIG. 5C, the pose angle θ mayalso be defined as an angle that is formed by rotating the calibrationboard 260 about the

_(Reference) axis and/or about the normal vector 261, which may be anaxis of rotation of the angle θ. Thus, in an embodiment, the pose anglesα, β, θ may represent respective amounts of rotation of the calibrationpattern 260 about respective axes of rotation (e.g.,

_(Reference),

_(Reference),

_(Reference)), wherein the respective axes are orthogonal to each other.In some instances, the respective axes may be parallel with ororthogonal to a camera optical axis, which may be an axis that isparallel with

_(Camera). In other instances, they may be oblique to the camera opticalaxis.

In an embodiment, the pose angles α, β, θ may be defined based ondifferent reference coordinate systems, which may have different sets ofcoordinate axes. For instance, FIGS. 6A-6C depict an example in whichthe pose angle β is defined relative to a reference axis that is basedon the pose angle α, and further depicts an example in which the poseangle θ is defined relative to another reference axis that is based onthe pose angles α and β. More specifically, FIG. 6A illustrates anembodiment in which, similar to the embodiment of FIG. 5A, the poseangle α is an angle formed between the normal vector 261 and a referenceaxis

_(Reference1) along a Y-Z plane formed by

_(Reference1) and

_(Reference1), wherein

_(Reference1),

_(Reference1), and

_(Reference1) may be coordinate axes of a first reference coordinatesystem. The normal vector 261 may be coincident with, or more generallyparallel with,

_(Pattern). The reference axis

_(Reference1) may be an axis that is coincident, or more generallyparallel, with the

_(Camera) of FIG. 4B. In some instances, the

_(Reference1) axis may be an optical axis of the camera 270. In someinstances,

_(Reference1) may be an axis that is parallel with the

_(Camera) axis of FIG. 4B, while

_(Reference1) axis may be an axis that is parallel with the

_(Camera) axis of FIG. 4B. In the example depicted in FIG. 6A, the poseangle α may also be defined as an angle between

_(Pattern) and

_(Reference1), and/or as an angle that is formed by rotating thecalibration board 260 about the

_(Reference1) axis.

In the example of FIGS. 6A-6C, the axes

_(Reference1),

_(Reference1),

_(Reference1) may be a first reference coordinate system, while the poseangle β may be defined relative to a second reference coordinate system.The second reference coordinate system may define a starting point forthe calibration board 260 before it is rotated by the pose angle β. Atsuch a starting point, the calibration board 260 has already beenrotated by the pose angle α relative to the first reference coordinatesystem. Thus, the second reference coordinate system in this example maybe a coordinate system that is rotated by α relative to the firstreference coordinate system. The second reference coordinate system maybe defined by the coordinate axes

_(Reference2),

_(Reference2),

_(Reference2). As depicted in FIG. 6B, the pose angle β may be an anglethat is formed between the normal vector 261 and

_(Reference2), or between

_(Pattern) and

_(Reference2), along a X-Z plane formed by

_(Reference2) and

_(Reference2). In the example depicted in FIG. 6B, the pose angle β mayalso be defined as an angle between

_(Pattern) and

_(Reference2), and/or as an angle that is formed by rotating thecalibration board 260 about the

_(Reference2) axis.

Similarly, the pose angle θ may be defined relative to a third referencecoordinate system. The third reference coordinate system may define astarting point for the calibration board 260 before it is rotated by theangle θ. This starting point may be defined by rotating the secondcoordinate system by the angle β, which may yield the coordinate axes

_(Reference3),

_(Reference3),

_(Reference3) for the third reference coordinate system. As illustratedin FIG. 6C, the pose angle θ may be an angle that is formed between

_(Pattern) and

_(Reference3), or between

_(Pattern) and

_(Reference3), along a X-Y plane formed by

_(Reference3) and

_(Reference3). In the example depicted in FIG. 6C, the pose angle θ mayalso be defined as an angle that is formed by rotating the calibrationboard 260 about the

_(Reference) axis, or about the normal vector 261.

As stated above, one aspect of the embodiments herein relate tocontrolling a calibration pattern 160/260 to have diverse poses, andmore specifically to adopting a plurality of pattern orientations thathave a desired distribution, such as a generally uniform distributionwithin a desired range of pattern orientations. The pattern orientationsmay be controlled by one or more pose angles, such as α, β, θ. However,simply generating angle values (also referred to as pose angle values)for each of the pose angles α, β, θ individually, according to a uniformdistribution, may not necessarily cause resulting pattern orientationsto have a uniform distribution.

For example, FIG. 7A depicts an example that represents patternorientations that each result from individually generating a random poseangle value for α based on a uniform probability density distribution(PDF), and generating a random pose angle value for β based on theuniform PDF. In some cases, a random pose angle value for θ may also beindividually generated according to the uniform PDF. More specifically,FIG. 7A depicts an imaginary sphere 302 that may represent possiblepattern orientations for the calibration pattern 160/260. A surface ofthe imaginary sphere may represent possible directions at which thenormal vector 261 of the calibration pattern 160/260 can be pointed.More specifically, the surface of the imaginary sphere 302 may be or mayrepresent a loci of all directions to which the normal vector 261 of thecalibration pattern 260 can be pointed. If the normal vector 261 has anassigned length (e.g., 10 cm), then the imaginary sphere 302 may have aradius equal to the assigned length. An example of the normal vector 261is depicted in FIG. 7B, and may be a vector that is orthogonal to thecalibration pattern 260. In an embodiment, the normal vector 261 mayhave one endpoint that is on the calibration pattern 260. For instance,the endpoint may be located at an origin of the pattern coordinatesystem for the calibration pattern 260. In the example of FIG. 7B, acenter of the imaginary sphere 302 may be at that endpoint of the normalvector 261, and a surface of the imaginary sphere 302 may be defined bythe other endpoint of the normal vector 261. A specific surface point onthe surface of the imaginary sphere 302 may represent a specific patternorientation of the calibration pattern 260 that would cause the normalvector of the calibration pattern 260 to point to that surface point.For instance, when the calibration pattern 260 has the particularorientation depicted in FIG. 7B, the normal vector 261 points to surfacepoint 304 a. Thus, the surface point 304 a may represent or otherwisecorrespond to that pattern orientation. When the calibration pattern 260changes orientation, the normal vector 261 may point to a differentsurface point on the imaginary sphere 302, such as surface point 304 bor 304 c, which are illustrated in FIG. 7A.

As stated above, the example in FIG. 7A may generate a random pose anglevalue for a based on a uniform probability density distribution (PDF),and generating a random pose angle value for β based on the uniform PDF.The uniform PDF may be configured to randomly output a value that is ina desired range of angle values, wherein each angle value in the rangehas an equal likelihood of being outputted. In the example of FIG. 7A,the desired range may be from −180 degrees to 180 degrees. The desiredrange of angle values may correspond with a desired range of patternorientations. For FIG. 7A, such a desired range of angle values (−180degrees to 180 degrees) may correspond to all possible patternorientations for the calibration pattern 260.

As depicted in FIG. 7A, while a uniform PDF is used to individuallydetermine pose angle values for each of the pose angles α, β, θ, theresulting pattern orientations are not uniformly distributed within adesired range of pattern orientations. More specifically, the desiredrange of pattern orientations in the example of FIG. 7A may include allpossible pattern orientations, and the resulting pattern orientationsfrom determining angle values using the uniform PDF are represented assurface points (e.g., 304 a-304 d) on a surface of an imaginary sphere302. The plurality of surface points in FIG. 7A are distributed moredensely within certain portions of the surface of the imaginary sphere302 than within other portions on the surface of the imaginary sphere302. Thus, the resulting pattern orientations may be more denselydistributed toward a certain range or ranges of directions relative toother directions. By contrast, FIG. 7C illustrates a scenario in whichpattern orientations have a more uniform distribution, such that surfacepoints representing the pattern orientations also have a more uniformdistribution on the surface of the imaginary sphere 302.

In FIG. 7A, the pattern orientations resulted from a desired range ofangle values for α, β, and/or θ that is from −180 degrees to 180degrees. FIGS. 8A-8C depict a distribution of pattern orientations basedon a different desired range of angle values. More specifically, FIG. 8Adepict an example in which pattern orientations result from a firstdesired range 802 (also referred to as the first range 802) of anglevalues that is from −10 degrees to −30 degrees, and a second desiredrange 804 (also referred to as the second range 804) of angle valuesthat is from 10 degrees to 30 degrees. A pose angle value for the poseangle α may be constrained to the first range and the second range. Thepose angle α may be constrained to the first range 802 and the secondrange 804 so that the calibration pattern 260 is tilted relative to thecamera 270 when a calibration image is acquired, but is not too tilted.In some cases, the desired range or ranges of angles for one or more ofα, β, θ may be one or more user-defined ranges.

In an embodiment, the desired range of angle values may apply tomultiple pose angles, such as to α and β. In such an embodiment, thepose angle β would also be constrained to the first range 802 and thesecond range 804 discussed above. In an embodiment, a region on thesurface of the imaginary sphere (which may also be referred to as asurface region) may represent a range or ranges of pattern orientationsresulting from the desired ranges 802/804 of angle values. For instance,FIG. 8B illustrates a surface region 306 of the imaginary sphere 302.The surface region 306 may contain surface points that represent thedesired range of pattern orientations. For instance, the patternorientation of the pattern 260 in the example depicted in FIG. 8B may berepresented by the surface point 304 d. In an embodiment, the surfaceregion 306 may form a circular or elliptical band on the surface of theimaginary sphere 302. In an embodiment, the circular or elliptical bandmay have uniform width.

In the example of FIG. 7A, a pose angle value for each of α and β may bedetermined based on a uniform probability distribution that randomlyselects an angle value in a desired range of angle values of −180degrees to 180 degrees. FIG. 8B depicts an example in which the poseangle value of α and β is determined in a similar way, but the desiredrange of angle values is the first range 802 and the second range 804 ofFIG. 8A, namely −10 degrees to −30 degrees, and 10 degrees to 30degrees. The pose angles α and β may be constrained to the first range802 and the second range 804, which may result in a desired range ofpattern orientations represented by the surface region 306. Determiningα and β in this way, however, also results in a distribution of patternorientations within the desired range of pattern orientations that isnot generally uniform, or that more generally does not have a desireddistribution. More specifically, FIG. 8C depicts a plurality of surfacepoints, such as surface points 306 a-306 c, that represent adistribution of pattern orientations resulting from determining poseangle values with the above technique. As the figure illustrates, theplurality of surface points are not distributed uniformly within thesurface region 306, but rather are concentrated near a top of thesurface region 306. This distribution of surface points may indicatethat the resulting pattern orientations may be more concentrated arounddirections that have closer alignment with

_(Reference), and that tend to exhibit less tilt relative to the camera170/270.

As stated above, one aspect of the embodiments herein relate todetermining a plurality of poses that have pattern orientations whichare distributed in a desired manner, such as a generally uniform mannerwithin a desired range of pattern orientations for the calibrationpattern 160/260. For instance, FIG. 9 depicts a method 900 fordetermining a plurality of poses that may have pattern orientationswhich are distributed in a desired manner. The method 900 may beperformed by the control circuit 111 of the computing system 110, aspart of performing camera calibration. The camera calibration can beused to, e.g., determine estimates for intrinsic camera parameters forthe camera 170, and/or determine a spatial relationship between thecamera 170 and its environment, such as a location and orientation ofthe camera 170 relative to the robot 150.

In an embodiment, the method 900 may include a step 902, in which thecontrol circuit 111 determines a range of pattern orientations, whichmay be a range of pattern orientations of the calibration pattern160/260 of FIG. 1/3A for performing camera calibration. In someinstances, the range of pattern orientations may be a desired range ofpattern orientations. In some instances, the range of patternorientations may include only some of the possible pattern orientationsthat can be adopted by the calibration pattern 160/260, and excludeother possible pattern orientations. The excluded pattern orientationsmay, e.g., provide calibration images that are not optimal forperforming camera calibration. For instance, a pattern orientation thatexhibits no tilt relative to the camera 170/270 of FIG. 1/3A may yield acalibration image which exhibits no perspective effect and/or no lensdistortion effect, which may have only limited value for determining anestimate of a lens distortion parameter or other camera calibrationparameter. Such a pattern orientation may correspond to a pose anglevalue of zero degrees for both the pose angle of α and β. Some otherpattern orientations may be excluded because they cause pattern elementsor other features on the calibration pattern 160/260 to face away toomuch from the camera 170/270, such that photographing the calibrationpattern 160/260 at those poses may yield calibration images that fail toclearly capture the pattern elements.

In an embodiment, determining the range of pattern orientations in step902 may involve determining one or more ranges of angle values for atleast one pose angle, such as the pose angles α, β, or θ discussedabove. The range that is determined for the pose angle may constrainwhich pose angle values can be determined for that pose angle. In oneexample, the one or more ranges may be the first range 802 and thesecond range 804 depicted in FIG. 8A. More specifically, determining therange of pattern orientations in such an example may involve determiningthat both the pose angle α and the pose angle β are constrained to tworanges, namely the first range 802 that is between −10 degrees and −30degrees, and the second range 804 that is between 10 degrees and 30degrees. In some instances, the one or more ranges may exclude poseangle values that are too small in magnitude (i.e., in absolute value)and may further exclude pose angle values that are too large inmagnitude. The pose angle values that are too small in magnitude maycause the calibration pattern 160/260 to exhibit no tilt or too littletilt relative to the camera 170/270, which may lead to calibrationimages that are not optimal for accurately estimating camera parametervalues. The pose angle values that are too large in magnitude may causethe calibration pattern 160/260 to be too tilted relative to the camera170/270, to a pattern orientation at which various pattern elements orother features on the calibration pattern 160/260 are not captured in aresulting calibration image, or appear too warped in the resultingcalibration image.

In some cases, the range of pattern orientations for step 902 may bebased on user-defined values. For instance, determining the range ofpattern orientations may involve the control circuit 111 of FIG. 2accessing a user-defined range or ranges of pose angle values from thenon-transitory computer-readable medium 115. More specifically, thecontrol circuit 111 may retrieve, or more generally receive, theuser-defined range from the non-transitory computer-readable medium 115.

In an embodiment, method 900 includes step 904, in which the controlcircuit 111 of FIG. 2 determines a surface region on a surface of animaginary sphere. As an example, the imaginary sphere may be theimaginary sphere 302 of FIGS. 7A-7C, 8B, 8C, and also of FIGS. 10A and10B. The imaginary sphere (e.g. imaginary sphere 302) may representpossible pattern orientations for the calibration pattern 160/260. In amore specific example, the imaginary sphere may represent all possiblepattern orientations for the calibration pattern 160/260. For instance,a surface of the imaginary sphere may be a loci of all points to which anormal vector (e.g., 261) of the calibration pattern 160/260 can point,and may correspond to all directions to which the normal vector canpoint. If the normal vector is assigned a defined length of, e.g., 10cm, then the imaginary sphere may be a sphere having a radius of 10 cm.In an embodiment, surface points which are outside of the surface regionare not used to determine pose angle values. In other words, the controlcircuit 111 may ignore surface points which are outside of the surfaceregion for purposes of determining pose angle values and determiningposes.

In an embodiment, the surface region (e.g. 306) on the surface of theimaginary sphere (e.g., 302) represents the range of patternorientations for performing the camera calibration (e.g., the desiredrange of pattern orientations for performing camera calibration). Forinstance, the surface region may be the surface region 306 of FIGS. 10Aand 8C. In some cases, the surface region may be a loci of points thatthe normal vector of the calibration pattern 160/260 can point to whilestaying within the desired range of pattern orientations. As an example,if the range of pattern orientations is based on one or more ranges ofpose angle values for at least one pose angle (e.g., α and β, thesurface region (e.g., 306) may be a loci of points to which the normalvector (e.g., normal vector 261) can point to while staying within theone or more ranges of pose angle values for the at least one pose angle.In some cases, the surface region, such as the surface region 306 ofFIGS. 10A and 8C, may form a circular band (also referred to as acircular strip), on the surface of the imaginary sphere. The circularband may have uniform width, or may have varying width.

In an embodiment, the method 900 includes a step 906, in which thecontrol circuit 111 determines a plurality of poses for the calibrationpattern 160/260. In some instances, the plurality of poses may be posesat which the calibration pattern 160/260 is photographed or otherwiseimaged to generate calibration images, and may be referred to asimage-captured poses or imaged poses. The plurality of poses may bedefined by respective combinations of a plurality of respectivelocations within the camera field of view and a plurality of respectivesets of pose angle values. For example, the plurality of respectivelocations may be locations within the camera field of view 272 of FIGS.3A, 3B, and 4B. In this example, each of the respective sets of poseangle values may be respective pose angle values for α and β, orrespective pose angle values for α, β, and θ. In such an example, aparticular pose may be defined by a location within the camera field ofview 272, a pose angle value for α, a pose angle value for β, and a poseangle value for θ. In some cases, the pose angles used to define a posemay be angles that affect how much the calibration pattern 160/260 istilted relative to the camera 170/270. Such angles may include the poseangles α and β. If, for instance, the pose angle θ does not affect therelative tilt of the calibration pattern 160/260, then the pose of thecalibration pattern 160/260 may be defined by only the pose angles α andβ and a location within the camera field of view 272.

In an embodiment, each set of pose angle values of the plurality of setsof pose angle values in step 906 may be determined based on a respectivesurface point selected from within the surface region on the surface ofthe imaginary sphere. For instance, the set of pose angle values mayinclude three angle values for the pose angles α, β, and θ,respectively, or include two angle values for the pose angles α and β,respectively. In this example, some or all of the pose angle values inthe set of pose angle values may be based on a respective surface point,such as one of surface points 308 a-308 i in FIG. 10A, selected fromwithin the surface region 306 on the surface of the imaginary sphere 302in the figure. The surface point (e.g., surface point 308 a) mayrepresent a direction at which the normal vector 261 of the calibrationpattern 160/260 is pointed. The determination of the pose angle valuesis discussed below in more detail.

In an embodiment, determining poses for the calibration pattern 160/260by selecting surface points on an imaginary sphere that representspossible pattern orientations for the calibration pattern 160/260, andthen determining pose angle values for at least one pose angle based onthe selected surface points may better allow the resulting patternorientations to achieve a desired distribution. For instance, thesurface points on which the respective set of pose angle values arebased may be randomly selected from within the surface region accordingto a uniform probability distribution, or some other probabilitydistribution (e.g., a Gaussian distribution). Using a uniformprobability distribution to select the surface points may ensure thatthe selected surface points are likely to have a uniform distributionwithin the surface region. In such an example, because the surfacepoints which are selected are likely to have a uniform distributionwithin the surface region, the pose angle values which are determinedbased on the selected surface points are also likely to yield resultingpattern orientations that have a uniform distribution or some otherdesired distribution.

FIG. 10A depicts an example of a plurality of surface points 308 a-308 ithat are selected from within the surface region 306 of the imaginarysphere 302, and which have a substantially uniform distribution withinthe surface region 306. More specifically, the surface region 306 mayform a circular band, and the surface points 308 a-308 i are distributedin a substantially uniform manner around the circular band, and along awidth of the circular band. As stated above, using the surface points308 a-308 i to determine the pose angle values for at least one poseangle may yield pattern orientations that have a substantially uniformdistribution.

In an embodiment, the control circuit 111 may be configured, forrespective surface points (e.g., 308 a-308 i) on which the respectivesets of pose angle values are based, to randomly select each of therespective surface points from within the surface region, such as thesurface region 306 in FIGS. 10A and 10B, according to a uniformprobability distribution. In some cases, the random selection may relyon a pseudorandom function, such as rand( ). In some cases, the surfaceregion may be defined in terms of a range of polar coordinates, and eachsurface point of the respective surface points may be selected byrandomly selecting a polar coordinate from among the range of polarcoordinates. The random selection may be performed according to auniform probability distribution such that each polar coordinate in therange of polar coordinates is equally likely to be selected. In somecases, the control circuit 111 may be configured, for the respectivesurface points on which the respective sets of pose angle values arebased, to randomly select each of the respective surface points fromamong only a uniform set of surface points. The uniform set of surfacepoints may be a set of surface points that are uniformly distributedwithin the surface region on the surface of the imaginary sphere. Forexample, FIGS. 10A and 10B depict a uniform set of surface points. Insuch an example, the plurality of surface points 308 a-308 i may besurface points that are randomly selected from among the uniform set ofsurface points. The random selection may be performed according to auniform probability distribution in which each of the uniform set ofsurface points is equally likely to be selected. By performing theselection in this manner, the surface points which are selected (e.g.,308 a-308 i) may tend to have a generally uniform distribution withinthe surface region 306.

As stated above, in some instances a surface point on the surface of theimaginary sphere (e.g., 302) may represent a respective orientation forthe calibration pattern 160/260 that would cause a normal vector (e.g.,261) for the calibration pattern to point to or otherwise be directedtoward the surface point. For example, FIG. 10B illustrates the surfacepoint 308 a that represents a respective orientation for the calibrationpattern 160/260 that would cause the normal vector 261 to point to thatsurface point 308 a. In such instances, determining a pose angle valuebased on the surface point may involve applying an arctangent torespective coordinates for the surface point 308 a. In this example, therespective coordinates for the surface point 308 a may also be referredto as coordinates of the normal vector 261. For instance, if the surfacepoint 308 a has a Cartesian coordinate of [x, y, z]^(T) (in the cameracoordinate system, or some other coordinate system), an angle value forone of the pose angles (e.g., α) may be equal to or based onarctan(y/z). In some implementations, the angle values may be determinedbased on solving for one or more rotation matrices which would transformthe normal vector from pointing in an initial direction (e.g., pointingalong a camera optical axis, toward a coordinate [0 0 10 cm]^(T)) topointing in a direction toward the surface point 308 a (e.g., toward thecoordinate [x y z] ^(T)). In one example, solving for the rotationmatrices may involve solving the equation [x y z]^(T)=R_(α)R_(β)R_(θ)[00 10 cm]^(T), wherein R_(α), R_(β), and R_(θ) are the respectiverotation matrices representing rotation of the calibration pattern160/260 with the pose angles α, β, and θ in the manner described abovewith respect to FIGS. 5A-5C and 6A-6C. In some implementations, if thesurface point 308 a has coordinates which are polar coordinatesexpressed in a polar coordinate system, the pose angle values for someof the pose angles may be based on, or more specifically equal to,components of the polar coordinates.

As stated above, in an embodiment the plurality of poses that aredetermined in step 906 may be the poses at which the calibration pattern160/260 is photographed or otherwise imaged by the camera 170/270 togenerate the calibration images for performing camera calibration. Thus,the plurality of poses determined in step 906 may also be referred to asimage-captured poses. In some implementations, determining the pluralityof poses in step 906 may involve determining a set of candidate poses,determining which of the candidate poses are robot-achievable candidateposes, and selecting the plurality of poses (which are theimage-captured poses) from among the robot-achievable candidate poses.

In an embodiment, a candidate pose may be a pose that the controlcircuit 111 has determined, but has not yet evaluated whether the posecan be achieved by the robot 150/250, as discussed below in more detail.In some cases, the candidate pose may be a pose for which the controlcircuit 111 has determined a location and a set of pose angle values.For example, each candidate pose of the set of candidate poses may bedetermined by: determining a respective location within the camera fieldof view for the candidate pose and determining a respective set of poseangle values for the candidate pose. The respective location may bedetermined, for instance, to result in robot-achievable candidate posesthat are spread out in space, as discussed below in more detail. In somecases, determining the respective location may rely on a function thatgenerates a random or pseudo-random value (e.g., a rand( ) function) forsome or all components of a coordinate for the respective location. Inan embodiment, the respective set of pose angle values may be determinedby, e.g., selecting a respective surface point from within the surfaceregion (e.g., 306) on the surface of the imaginary sphere (e.g., 302),and determining the respective set of pose angle values for thecandidate pose based on the respective surface point, as discussedabove. In another embodiment, the respective set of pose angle valuesmay be determined in a different manner.

In an embodiment, the control circuit 111 may be configured todetermine, from the set of candidate poses, a set of robot-achievablecandidate poses. A robot-achievable candidate pose may be a candidatepose for the calibration pattern 160/260 that can be can be achieved bythe robot 150/250. More specifically, the robot 150/250 may in somescenarios be unable to achieve some candidate poses. For example, aparticular candidate pose may have a set of the pose angle values thatthe robot 150/250 is unable to fulfill because the robot 150/250 isunable to tilt the calibration pattern in a manner indicated by that setof the pose angle values. Additionally, the candidate pose for thecalibration pattern may involve not only the set of pose angle values atwhich to place the calibration pattern 160/260, but also a locationwithin the camera field of view (e.g., 272) at which to place thecalibration pattern 160/260. In some instances, the robot 150/250 may beunable to place the calibration pattern 160/260 to the determinedlocation. In some instances, the robot 150/250 may be able to fulfilleither the set of pose angle values or the location of the candidatepose, but may be unable to fulfill a combination of both the set of poseangles and the location of the candidate pose, because of constraints onthe movement of the robot 150/250. For example, movement of the robot150/250 may be constrained by obstacles, which may prevent the robot150/250 from moving the calibration pattern 160/260 to certain locationin the camera field of view (e.g., 272). In some instances, a mechanicalconfiguration of the robot 150/250 may constrain its freedom ofmovement. As an example, the robot 250 of FIGS. 3A and 3B may have amechanical configuration in which various links 254A-254E of a robot armare connected to each other, and have limited degrees of freedomrelative to each other. Such a mechanical configuration may prevent therobot 250 from achieving certain combinations of location andorientation for the link 254E, to which the calibration pattern 260 isattached. Thus, the mechanical configuration of the robot 250 mayprevent the robot 250 from achieving certain combinations of locationand pattern orientation of the calibration pattern 260. In other words,the mechanical configuration of the robot 250 may prevent the robot 250from achieving certain poses for the calibration pattern 260.

Thus, in an embodiment, the control circuit 111 in step 906 maydetermine, for each candidate pose of the set of candidate poses,whether the candidate pose is robot-achievable (i.e., whether thecandidate pose is able to be achieved by the robot 150/250). The controlcircuit 111 may, in response to a determination that the candidate poseis robot-achievable, add the candidate pose to the set of candidateposes. The control circuit 111 further may, in response to adetermination that the candidate pose is not robot-achievable, excludethe candidate pose from the set of robot-achievable candidate poses, ormore generally ignore the candidate pose for purposes of performingcamera calibration.

In some cases, the control circuit 111 may determine whether aparticular candidate pose is robot-achievable by controlling the robot150/250 to actually attempt to move the calibration pattern 160/260 toachieve the candidate pose, and determining whether the robot 150/250 isable to achieve the candidate pose within a defined amount of time. Insome cases, the control circuit 111 may determine whether an inversekinematics function is able output a movement command for the candidatepose. The inverse kinematics function may be a function that is designedto calculate a movement command, such as one or more motor commands, forthe robot 150/250 to accomplish a particular pose. If the inversekinematics function is able to output a movement command for theparticular candidate pose, the control circuit 111 may determine thatthe candidate pose is a robot-achievable candidate pose. If the functionis unable to output a movement command for the particular candidatepose, the control circuit 111 may determine that the candidate pose isnot a robot-achievable candidate pose.

As stated above, in an embodiment the control circuit 111 in step 906may further select the plurality of poses (which are or will be theimage-captured poses) from among only the set of robot-achievablecandidate poses. In some cases, the selection may involve selecting atarget number of robot-achievable candidate poses as the plurality ofposes. The target number may be, e.g., a user-defined value or may bedetermined based on some noise level, an amount of time allotted toperform camera calibration, or some other factor. For example, the setof robot-achievable candidate poses may include at least ninerobot-achievable candidate poses, and the target number may be eight. Insuch an example, the control circuit 111 in step 906 may select, as theplurality of poses, eight robot-achievable candidate poses from amongthe set of nine robot- achievable candidate poses. In another example,the set of robot-achievable candidate poses may include at least 64candidate poses, and the target number may be 15. In such an example,the control circuit 111 may select, as the plurality of poses, 15robot-achievable candidate poses from among the set of 64robot-achievable candidate poses. In some implementations, the controlcircuit 111 may perform the selection randomly. For instance, thecontrol circuit 111 may randomly select the 15 robot-achievablecandidate poses from among the set of 64 robot-achievable candidateposes according to a uniform probability distribution in which each ofthe robot-achievable candidate poses are equally likely to be selected.The random selection may, in some implementations, rely on apseudorandom function.

As stated above, the control circuit 111 may in an embodiment todetermine a respective set of pose angle values (e.g., for respectivepose angles α, β, θ) for each of the set of candidate poses based on asurface point selected from within a surface region (e.g., 306) on asurface of an imaginary sphere (e.g., 302). Because the plurality ofposes determined in step 906 (which are or will be the image-capturedposes) are ultimately selected from the set of candidate poses, each ofthe plurality of poses may be considered to have a set of pose anglevalues that are also determined based on a respective surface pointselected from within the surface region on the surface of the imaginarysphere.

In an embodiment, the control circuit 111 may determine respectivelocations for candidate poses in a random manner. For instance, thecontrol circuit 111 may randomly select a location that is within thecamera field of view (e.g., 272), and determine a set of pose anglevalues based on a surface point selected from within a surface region ofan imaginary sphere (in the manner described above), and evaluatewhether a candidate pose having the determined location and set of poseangle values is a robot-achievable candidate pose. In some cases, theset of pose angle values may be determined in some other manner thatdoes not rely on determining surface points. In an embodiment, thecontrol circuit 111 may determine locations for candidate poses in amanner such that the candidate poses are spread out within the camerafield of view. More specifically, the control circuit 111 may determinelocations for candidate poses such that those candidate poses result inrobot-achievable candidate poses that are spread out within the camerafield of view. Because the plurality of poses determined in step 906 maybe selected from the robot-achievable candidate poses, the plurality ofposes may then also be spread out within the camera field of view.

In an embodiment, to attempt to spread out the candidate poses,robot-achievable candidate poses, and/or the image-captured poses, thecontrol circuit 111 may determine a grid of 3D regions that divide aspace within the camera field of view (e.g., 272), and determinelocations for the candidate poses such that they are spread out in thegrid, and/or such that the robot-achievable candidate poses are spreadout in the grid. In an embodiment, the grid of 3D regions may divide aspace within the camera field of view into one or more layers that eachhas multiple rows of 3D regions and multiple columns of 3D regions.

In an embodiment, the space within the camera field of view may be aspace in which the calibration pattern 160/260 is moved by the robot150/250 and photographed by the camera 270 to perform cameracalibration. The space may be large enough to include all locationswithin the camera field of view (e.g., 272) to which the robot 150/250can move the calibration pattern 160/260, or may have a size that leavesout some of those locations from the space. In some cases, the size orboundaries of the space may be based on a range of motion of the robot150/250. For instance, the boundaries of the space may correspond to thefarthest locations that the robot 150/250 (e.g., via a robot arm) isable to place the calibration pattern 160/260 relative to a base (e.g.,252) of the robot, or relative to the camera 170/270, or relative tosome other location. In some instances, the boundaries of the space maybe defined by a first depth value and a second depth value. Forinstance, FIG. 11A depicts a space 271 within the camera field of view272 of the camera 270. In this example, the space 271 may enclose alllocations (and only those locations) that are within the camera field ofview 272 and that are between a first depth value Depth_(min) and asecond depth value Depth_(max), wherein both depth values are relativeto the camera 270. In some cases, the first depth value and the seconddepth value may be user-defined values, which are stored in thenon-transitory computer-readable medium 115 of FIG. 2 or in some otherdevice, and are accessible to the control circuit 111. In some cases,the control circuit 111 may determine the first depth value and thesecond depth value based on the range of motion of the robot 150/250. Inan embodiment, the space 271 may form or may enclose a frustum of apyramid or a cone. The pyramid or cone may define the camera field ofview. For instance, the field of view 272 in FIG. 11A is defined by apyramid, and the space 271 may form a frustum of the pyramid.

As stated above, the grid of 3D regions may divide the space within thecamera field of view into one or more layers that each has multiple rowsof 3D regions and multiple columns of 3D regions. For instance, the FIG.11A depicts a grid of twenty-seven 3D regions 273 ₁₋₂₇ (i.e., 3D region273 ₁, 273 ₂, 273 ₃, . . . 273 ₂₅, 273 ₂₆, 273 ₂₇) that divide the space271 into a first layer 274, a second layer 275, and a third layer 276.The grid illustrated in FIG. 11B may be a 3×3×3 grid. That is, the gridmay have three layers (the first layer 274, second layer 275, and thirdlayer 275), and each layer may have three rows of 3D regions and threecolumns of 3D regions. In other words, each of the layers 274, 275, and276 may be or may be divided into a 3×3 grid of three columns and threerows. The first layer 274 may contain 3D regions 273 ₁₋₉, the secondlayer 275 may contain 3D regions 273 ₁₀₋₁₈, and the third layer 276 maycontain 3D regions 273 ₁₉₋₂₇.

In the example of FIG. 11B, in which the field of view 272 is defined bya pyramid, each 3D region may form or be shaped as a hexahedron. In somecases, the hexahedrons may be cubes. In another example in which acamera field of view is defined by another shape, such as a cone, someor all of the 3D regions may have different shapes. In an embodiment,the 3D regions 273 ₁₋₂₇ collectively may completely occupy all of thespace 271, and may be non-overlapping regions. In an embodiment, each ofthe 3D regions 273 ₁₋₂₇ may be immediately adjacent to other ones of the3D regions, such that the 3D region shares a boundary with some of theother 3D regions. For instance, as depicted in FIG. 11A, 3D region 273 ₁shares a boundary with two other 3D regions in the first layer 274, andshares a boundary with another 3D region on the second layer 275.

In an embodiment, the control circuit 111 may be configured to determinea target number that indicates how many poses are desired for theplurality of poses in step 906, and may determine a size of the gridbased on the target number. The size may indicate how many layers, rows,and/or columns are in the grid, which may affect how many 3D regions arein the grid. In some cases, the control circuit 111 may determine thegrid size as a smallest integer that is greater than or equal to asquare root of the target number. More specifically, the grid may haveone or more layers, and have n rows per layer, and n columns per row. Insome situations, the grid may be able to contain at most nrobot-achievable candidate poses per layer, such as in examples in whichthe robot-achievable candidate poses have to satisfy a Latin squarespatial distribution, or a stratified spatial distribution, as discussedbelow in more detail. If the grid further has n layers (i.e., the gridis a n×n×n grid), then the grid may be able to accommodate at mostcontain at most n² robot-achievable candidate poses in the abovesituations. Because the plurality of poses in step 906 may be selectedfrom among the set of robot-achievable candidate poses, the n²robot-achievable candidate poses need to be greater in quantity than thetarget number, which indicates how many poses are to be determined forthe plurality of poses in step 906. Thus, the control circuit 111 may beconfigured to determine, as the size of the grid, a value of n as asmallest integer which is greater than or equal to a square root of thetarget number of poses. Such a value for n may ensure that the number ofrobot-achievable candidate poses in the above situation, which is equalto n², is greater than the target number determined for step 906. Thesize n that is determined may indicate how many rows are in the grid,how many columns are in the grid, how many layers in the grid, anycombination thereof, or may indicate some other information.

As stated above, the control circuit 111 may determine respectivelocations for candidate poses such that the candidate poses, or morespecifically a subset of the candidate poses that are robot-achievablecandidate poses, are spread out within the grid of 3D regions. Becausethe plurality of poses determined in step 906 (which may be referred toas image-captured poses) are selected from among the robot-achievablecandidate poses, the poses determined in step 906 may also be spread outwithin the grid of 3D regions. In an embodiment, the candidateposes/robot-achievable candidate poses/image-captured poses may bespread out within each layer of the 3D grid. For instance, they may bespread out within the first layer 274 of the grid of FIG. 11B, spreadout within the second layer 275 of the grid, and spread out within thethird layer 276 of the grid.

In some implementations, as discussed below in more detail, the controlcircuit 111 may attempt to find candidate poses to fill every 3D regionof the grid of 3D regions with exactly one candidate pose that is arobot-achievable candidate pose (or, more generally, to fill every 3Dregion with an equal number of candidate poses that are alsorobot-achievable candidate poses). In some implementations, as alsodiscussed below in more detail, the control circuit 111 may determinelocations for the candidate poses in an attempt to fill only a subset of3D regions with candidate poses, or more specifically with candidateposes that are robot-achievable candidate poses. In theseimplementations, the control circuit 111 may determine the locationssuch that the robot-achievable candidate poses in a particular layerhave a particular spatial distribution, such as a Latin hypercubespatial distribution (also referred to as a Latin square spatialdistribution), a stratified spatial distribution, or some otherdistribution, as discussed below in more detail.

As stated above, in an embodiment the control circuit 111 may determinerespective locations for candidate poses in an attempt to fill every 3Dregion of the grid of 3D regions (e.g., 273 ₁₋₂₇) with an equal numberof candidate poses (e.g., with exactly one pose), or more specificallywith an equal number of candidate poses that are also robot-achievablecandidate poses. In such an embodiment, the robot-achievable candidateposes may thus have a spatial distribution that is generally uniform. Insome cases, the plurality of poses determined in step 906 (theimage-captured poses) may include all of those robot-achievablecandidate poses, or may be a randomly selected subset of all of therobot-achievable candidate poses. However, it may be difficult find, forevery 3D region of the grid of 3D regions, a candidate poses that isalso a robot-achievable candidate pose. For instance, as discussedabove, some 3D regions may have obstacles that impede movement of therobot 150/250 and of the calibration pattern 160/260 into that 3Dregion. In some instances, each candidate pose may include not only alocation that is within a particular 3D region, but also a set of poseangle values. The pose angle values may be determined based on a surfacepoint of an imaginary sphere, as discussed above, or in some othermanner. The robot 150/250 may be able to place the calibration pattern160/260 at that location, but may be unable to also tilt the calibrationpattern 160/260 to fulfill the set of pose angle values, and thus may beunable to achieve that candidate pose.

Thus, in some cases, the control circuit 111 may determine respectivelocations for candidate poses so as to fill only a subset of 3D regionsof a grid layer with robot-achievable candidate poses. In someinstances, the control circuit 111 may determine these locations to fillonly the subset of 3D regions in response to a determination that it isunable to find robot-achievable candidate poses to fill every 3D regionof the layer, or more specifically that it is unable to find suchrobot-achievable candidate poses within a defined amount of time. Insome instances, the control circuit 111 may determine the locations tofill only the subset of 3D regions without attempting to find,beforehand, robot-achievable candidate poses to fill every 3D region ofthe layer.

In an embodiment, the control circuit 111 may determine respectivelocations for the candidate poses so as to attempt to identifyrobot-achievable candidate poses with a spatial distribution that isspread out within a layer of the grid. In some cases, the controlcircuit 111 may determine locations for candidate poses such that theyresult in robot-achievable candidate poses having a Latin square spatialdistribution (also referred to as a Latin hypercube spatialdistribution). A Latin square spatial distribution or Latin hypercubespatial distribution for robot-achievable candidate poses may be aspatial distribution in which each row of the multiple rows within thelayer includes exactly one robot-achievable candidate pose, and eachcolumn of the multiple columns within the layer includes exactly onerobot-achievable candidate pose. In a more specific example, if the griddiscussed above has one or more layers that each has n rows of 3Dregions and n columns of 3D regions, the control circuit 111 maydetermine a set of robot-achievable candidate poses by determining, foreach layer of the one or more layers, a respective subset of nrobot-achievable candidate poses based on an initial condition that then robot-achievable candidate poses have n locations with a first spatialdistribution in which each row (of the n rows of the layer) includesonly one robot-achievable candidate pose, and each column (of the ncolumns of the layer) includes only one robot-achievable candidate pose.In some cases, the respective subset of robot-achievable candidate posesmay further have n sets of pose angle values that are based on nrespective surface points selected from the surface region (e.g., 306)on the surface of the imaginary sphere (e.g., 302).

For instance, FIG. 12A depicts an example of a Latin square spatialdistribution for three robot-achievable candidate poses in layer 274 ofthe grid depicted in FIGS. 11A and 11B. In FIG. 12A, the threerobot-achievable candidate poses are represented by X's. Morespecifically, the three robot-achievable candidate poses include a firstpose that is at a location which occupies row 1, column 1 (or, morespecifically, within a 3D region that occupies row 1, column 1); includea second pose that is at a location which occupies row 3, column 2; andinclude a third pose which occupies row 2, column 3. In the example ofFIG. 12A, each row of the multiple rows of 3D regions within the layer274 includes exactly one robot-achievable candidate pose, and eachcolumn of the multiple columns of 3D regions within the layer 274includes exactly one robot-achievable candidate pose. This spatialdistribution may cause the robot-achievable candidate poses to be spreadout within the camera field of view 272.

As stated above, the poses that are determined in step 906 (i.e., theimage-captured poses) may be selected from the robot-achievablecandidate poses. Thus, in an embodiment, if the robot-achievablecandidate poses have a Latin square spatial distribution, then theplurality of poses may have a spatial distribution in which each row ofthe multiple rows within the layer includes no more than one pose of theplurality of poses, and each column of the multiple columns within thelayer includes no more than one pose of the plurality of poses. Forexample, FIG. 12B depicts an example in which the plurality of posesdetermined in step 906 include the first robot-achievable candidate poseof FIG. 12A (in the 3D region occupying row 1, column 1), and the thirdrobot- achievable candidate pose of FIG. 12A (in the 3D region occupyingrow 3, column 2). In this example, row 1 and row 3 of the grid includesexactly one pose of the plurality of poses, while row 2 includes no poseof the plurality of poses. Additionally, column 1 and column 2 of thegrid includes exactly one pose of the plurality of poses, while column 3includes no pose of the plurality of poses.

In an embodiment, the control circuit may 111 may attempt to achieve aLatin square spatial distribution by controlling how respectivelocations are determined for the candidate poses. Generally speaking,when the control circuit 111 is determining a location for a particularcandidate pose, it may avoid placing the candidate pose in a 3D regionthat already contains a previously identified robot-achievable candidatepose, and avoid placing the candidate pose in a 3D region that shares arow or column with a previously identified robot-achievable candidatepose. More specifically, the control circuit 111 may be configured todetermine a respective location for each candidate pose of the set ofcandidate poses to be a location which is in a layer of the one or morelayers of the grid and which i) does not share a row with anyrobot-achievable candidate pose of the set of robot-achievable candidateposes in that layer, and ii) does not share a column with anyrobot-achievable candidate pose of the set of robot-achievable candidateposes in that layer.

For instance, FIG. 12C depicts an example in the control circuit 111 maydetermine a first candidate pose by determining a first location and afirst set of pose angle values for the first candidate pose. In thisexample, the control circuit 111 may have not yet identified anyrobot-achievable candidate pose in the first layer 274 when it isdetermining the first candidate pose. Thus, the first candidate pose canbe placed in any 3D region in the first layer 274. In some cases, thecontrol circuit 111 may determine the first location in a random mannerusing, e.g., a pseudorandom function. In the example of FIG. 12C, thefirst location may be in row 2, column 3. Further in this example, thefirst candidate pose may be determined as a robot-achievable candidatepose.

Further in FIG. 12C, the control circuit 111 may further determine asecond candidate pose by determining a second location and a second setof pose angle values for the second candidate pose. In this example,because there is a robot-achievable candidate pose in the 3D region atrow 2, column 3, the control circuit may select, for the secondlocation, a 3D region that is in not in row 2, and not in column 3. Inthe example of FIG. 12C, the control circuit 111 may select a 3D regionthat is in row 1, column 2. In some cases, the control circuit 111 maydetermine the second location by randomly selecting a location withinthe 3D region occupying row 1, column 3. The control circuit 111 mayfurther determine, however, that the second candidate pose is not arobot-achievable candidate pose. The control circuit 111 in this examplemay similarly determine a third candidate pose by determining a thirdlocation and a third set of pose angle values for the third candidatepose. For instance, the control circuit 111 may select a 3D regionoccupying row 3, column 2 of the grid, and determine the third locationby randomly selecting a location within that 3D region. The thirdcandidate pose in this example may be determined as a robot-achievablecandidate pose. Further, the control circuit 111 may then determine afourth candidate pose by determining a fourth location and a fourth setof pose angle values. Because a first 3D region at row 2, column 3 has arobot-achievable candidate pose, and because another 3D region at row 3,column 2 has another robot-achievable candidate pose, the controlcircuit 111 may be limited to determining the fourth location as alocation within the 3D region at row 1, column 1.

In an embodiment, when the set of robot-achievable candidate posesalready includes one or more robot-achievable candidate poses, if thecontrol circuit 111 is unable to identify another robot-achievablecandidate poses to satisfy the Latin square spatial distribution, eithergenerally or within a defined amount of time, it may delete some or allof the set robot-achievable candidate poses. The control circuit 111 maythen retry attempting to identify robot-achievable candidate poses thatcan satisfy the Latin square spatial distribution. For instance, if thecontrol circuit 111 in the example of FIG. 12C determines that thefourth candidate pose is not a robot-achievable candidate pose, and isfurther unable to identify a robot-achievable candidate pose in the 3Dregion at row 1, column 1, then the control circuit 111 may remove therobot-achievable candidate pose at row 2, column 3, and/or remove therobot-achievable candidate pose at row 3, column 2 from the set ofrobot-achievable candidate poses. The control circuit 111 may thengenerate additional candidate poses in an attempt to findrobot-achievable candidate poses to satisfy the Latin square spatialdistribution. In some cases, if the control circuit 111 is still unableto identify robot-achievable candidate poses that satisfy the Latinsquare spatial distribution, either generally or within a defined amountof time, it may attempt to identify robot-achievable candidate posesthat satisfy a stratified spatial distribution, as discussed below inmore detail.

In an embodiment, the control circuit 111 may determine locations forthe candidate poses such that they result in robot-achievable candidateposes with a stratified spatial distribution. In some cases, the controlcircuit 111 may use the stratified spatial distribution in response to adetermination that the initial condition discussed above, whichdescribes the Latin square distribution, cannot be satisfied. Forinstance, in the above example involving a n×n×n grid, the controlcircuit may determine, for each layer of the n layers of the grid,whether n robot-achievable candidate poses for the layer aredeterminable if the n robot-achievable candidate poses have to satisfythe initial condition. For instance, the control circuit may determinewhether, before a defined time limit expires or other constraint, it hassuccessfully found n robot-achievable candidate poses that satisfy theinitial condition. In some cases, as discussed above, therobot-achievable candidate poses may have respective orientations thatare determined based on surface points selected from a surface region ofan imaginary sphere (e.g., a selection that is based on a uniformprobability distribution). In such cases, the control circuit would bedetermining whether it can successfully find n robot-achievablecandidate poses having both a spatial distribution of the initialcondition and respective orientations determined using the surfacepoints of the imaginary sphere. In some circumstances, the controlcircuit may determine that, for a particular layer of the grid, that nrobot-achievable candidate poses are not determinable if they have tosatisfy the initial condition (e.g., that n robot-achievable candidateposes has not been successfully found that satisfy the initial conditionfor the layer before a defined time limit expired, or before some otherdefined constraint). In some cases, the control circuit 111 may use thestratified spatial distribution without attempting beforehand to findcandidate poses to satisfy a Latin square spatial distribution, andwithout determining whether it can find robot-achievable candidate posesthat satisfy the Latin square spatial distribution. The stratifiedspatial distribution for robot-achievable candidate poses may be aspatial distribution in which, for a particular layer of the grid of 3Dregions, (i) each row of the multiple rows of 3D regions within thelayer includes exactly one robot-achievable candidate pose, or (ii) eachcolumn of the multiple columns within the layer includes exactly onerobot-achievable candidate pose (wherein “or” generally is used hereinto refer to “and/or”). In the above example involving the n×n×n grid,the control circuit 111 may attempt to achieve a stratified spatialdistribution by determining, for each layer of the grid, a respectivesubset of n robot-achieve candidate poses based on a second condition inwhich the n robot-achievable poses have n locations in which each row(of the multiple rows of the layer) includes only one robot-achievablecandidate pose, or each column (of the multiple columns of the layer)includes only one robot-achievable candidate pose. In some cases, the nrobot-achievable candidate poses may have n sets of pose angles that arebased on respective surface points selected from the surface region onthe surface of the imaginary sphere.

For instance, FIG. 13A depicts an example of a stratified spatialdistribution for three robot-achievable candidate poses in layer 275 ofthe grid depicted in FIGS. 11A and 11B. In the example of FIG. 13A, thethree robot-achievable candidate poses are represented by X's. Morespecifically, the three robot-achievable candidate poses include a firstpose that is at a location which occupies row 1, column 3 (or, morespecifically, within a 3D region that occupies row 1, column 1); asecond pose that is at a location which occupies row 2, column 1; and athird pose which occupies row 3, column 3. Although each column does notcontain exactly one robot-achievable candidate pose (column 3 includestwo robot-achievable candidate poses occupying two respective 3D regionsin the column), this example still satisfies the stratified spatialdistribution, because each row contains includes exactly onerobot-achievable candidate pose.

In an embodiment, if the robot-achievable candidate poses have astratified spatial distribution, then the plurality of poses determinedin step 906 may have a spatial distribution in which each row of themultiple rows within the layer includes no more than one pose of theplurality of poses, or each column of the multiple columns within thelayer includes no more than one pose of the plurality of poses. Forexample, FIG. 13B depicts an example in which the plurality of posesdetermined in step 906 (the image-captured poses) include the firstrobot-achievable candidate pose of FIG. 13A (in the 3D region occupyingrow 1, column 3), and the third robot- achievable candidate pose of FIG.13A (in the 3D region occupying row 3, column 3). In this example, whilecolumn 3 of the grid includes two poses of the plurality of poses, row 1and row 3 of the grid includes exactly one pose of the plurality ofposes, while row 2 includes no pose of the plurality of poses.

In an embodiment, the control circuit 111 may attempt to achieve thestratified spatial distribution by controlling locations of thecandidate poses. For instance, the control circuit 111 may be configuredto determine a respective location for each candidate pose of the set ofcandidate poses to be a location which is in a layer of the one or morelayers of the grid and which i) does not share a row with anyrobot-achievable candidate pose of the set of robot-achievable candidateposes in that layer, or ii) does not share a column with anyrobot-achievable candidate pose of the set of robot-achievable candidateposes in that layer.

In an embodiment, the control circuit 111 may determine locations forthe candidate poses such that they result in robot-achievable candidateposes with any random spatial distribution. In some cases, the controlcircuit 111 may use any random spatial distribution for therobot-achievable candidate poses in response to a determination that itcannot find enough robot-achievable candidate poses to satisfy a Latinsquare spatial distribution, and cannot find enough robot-achievablecandidate poses to satisfy a stratified spatial distribution. In somecases, the control circuit 111 may use any random spatial distributionfor the robot-achievable candidate poses without attempting to find,beforehand, robot-achievable candidate poses to satisfy a Latin squarespatial distribution, and/or without attempting to find, beforehand,robot-achievable candidate poses to satisfy a stratified spatialdistribution. In the above example involving the n×n×n grid, the controlcircuit 111 may be configured to determine that n robot-achievablecandidate poses are not determinable if they have to satisfy the initialcondition, and/or that n robot-achievable candidate poses are notdeterminable if they have to satisfy the second condition. For instance,the control circuit may determine that it has not successfully found,within a defined time limit, n robot-achievable candidate poses thatsatisfy the initial condition for a layer of the grid, and/or hasdetermined that it has not successfully found, within the defined timelimit, n robot-achievable candidate poses that satisfy the secondcondition for the layer. The initial condition is associated with aLatin square spatial distribution, and the second condition isassociated with a stratified spatial distribution. In other words, thecontrol circuit 111 may be unable to find n robot- achievable candidateposes that satisfy the Latin square spatial distribution and thestratified spatial distribution. In such a situation, the controlcircuit 111 may perform the following for that layer of the grid:determining the respective subset of n robot-achievable candidate posesfor that layer based on a third condition in which the nrobot-achievable candidate poses have: (a) n locations that are randomlydistributed within n respective 3D regions of the layer. In some cases,the n robot-achievable candidate poses may have n sets of pose anglevalues that are based on n respective surface points selected from thesurface region non the surface of the imaginary sphere.

FIG. 14A depicts an example of three robot-achievable candidate poseswhose locations were randomly determined to occupy three different 3Dregions of the layer 276 of the grid depicted in FIGS. 11A and 11B. FIG.14B depicts an example of two image-captured poses that are selectedfrom among the robot-achievable candidate poses of FIG. 14A.

The above discussion of the Latin square spatial distribution and thestratified spatial distribution involve a grid having layers withmultiple rows and multiple columns, and each row containing exactly onerobot-achievable candidate pose, and/or each column containing exactlyone robot-achievable candidate pose. In an embodiment, the Latin squarespatial distribution and the stratified spatial distribution may moregenerally involve each row having an equal number of robot-achievablecandidate poses as the other rows, and/or each column having an equalnumber of robot-achievable candidate poses as the other columns. Forexample, the control circuit 111 may in some situations identifyrobot-achievable candidate poses such that each row within a particularlayer of a grid has exactly two robot-achievable candidate poses, andeach column within the layer has exactly two robot-achievable candidateposes.

In an embodiment, the control circuit 111 may be configured to performthe determination of whether a particular spatial distribution is beingsatisfied on a layer-by-layer basis. For instance, when the controlcircuit 111 determines a location for a particular candidate pose,wherein the location is within a particular 3D region within aparticular layer of a grid (e.g., the grid in FIGS. 11A and 11B), thecontrol circuit 111 may evaluate whether the Latin square spatialdistribution or the stratified spatial distribution is being satisfiedby comparing the location of the candidate pose with locations ofexisting robot-achievable candidate poses to evaluate whether thecandidate pose is in the same row or is in the same column as one of therobot-achievable candidate poses. However, the control circuit 111 maymore specifically compare the location of the candidate pose withrespective locations of only those robot-achievable candidate poses thatare in the same layer of the grid, so as to determine whether thecandidate pose will be in the same row or the same column asrobot-achievable candidate poses in that layer. An example of thelayer-by-layer determination is illustrated in FIG. 15A, which depicts agrid having robot-achievable candidate poses that satisfy a Latin squarespatial distribution for each layer of a first layer 274, a second layer275, and a third layer 276. The control circuit 111 may determine thespatial distribution depicted in FIG. 15A even though a first pair ofrobot-achievable candidate poses are in respective 3D regions that havethe same row and the same column. More specifically, one of therobot-achievable candidate poses is in a 3D region that is at row 1,column 1 in the layer 274, and another one of the robot-achievablecandidate poses is in another 3D region that is also at row 1, column 1,but in layer 276. However, because the control circuit 111 in thisexample performs the evaluation of whether a particular spatialdistribution is satisfied on a layer-by-layer basis, therobot-achievable candidate poses in FIG. 15A may still be considered tosatisfy a Latin square spatial distribution for each of the layers 274,275, 276.

In an embodiment, the control circuit 111 may be configured to allowdifferent layers of the grid to have different spatial distributions.For instance, FIG. 15B depicts an example in which the control circuit111 has identified three robot-achievable candidate poses that have aLatin square spatial distribution for a first layer 274 of the grid, hasidentified another three robot- achievable candidate poses that satisfya stratified spatial distribution for a second layer 275 of the grid,and has identified yet another three robot-achievable candidate posesthat satisfy a random spatial distribution for a third layer 276 of thegrid. In some cases, the control circuit 111 may have determinedlocations to satisfy a stratified spatial distribution for therobot-achievable candidate poses in the second layer 275 after beingunable to successfully find three robot-achievable candidate poses thatcan satisfy the Latin square spatial distribution in the second layer275.

In an embodiment, the control circuit 111 may apply a more stringentcondition for satisfying a Latin square spatial distribution. The morestringent condition may involve a space which is divided into a gridhaving m layers, wherein each layer has n rows and n columns. The numberof layers may be the same as the number of rows or columns, or may bedifferent as the number of rows or columns. For each layer of the mlayers, each row may have only one robot-achievable candidate pose, andeach column may have only one robot-achievable candidate pose. Underthis more stringent condition, each stack in the grid may have onlyrobot-achievable candidate pose. A stack may refer to m 3D regions ofthe grid that are on different respective layers of the grid and thathave occupy the same row and the same column within the respectivelayers. FIG. 15C depicts an example of nine robot-achievable candidateposes that satisfy the more stringent condition discussed above.

As stated above, the plurality of poses that are determined in step 906may be selected from robot-achievable candidate poses that aredistributed within a grid of 3D regions that divide a space within acamera field of view (e.g., 272). A total number of robot-achievablecandidate poses that are selected may be equal to the target numberdiscussed above. The plurality of poses may be used to generate aplurality of calibration images, wherein a total number of calibrationimages is also equal to the target number discussed above. As anexample, FIGS. 15A-15C illustrate situations in which the controlcircuit 111 has identified nine robot-achievable candidate poses thatare distributed within a camera field of view. In this example, if thetarget number is equal to, e.g., eight, then step 906 may involveselecting eight poses from among the nine robot-achievable candidateposes. In an embodiment, the selection may be done in a random manner,such as through the use of a pseudorandom function.

As the above discussion indicates, step 906 may involve determining aplurality of poses by determining a plurality of respective sets of poseangle values, wherein each set of pose angle values is determined basedon a respective surface point selected from within a surface region on asurface of an imaginary sphere. In some cases, step 906 may furtherinvolve determining locations for the plurality of poses to attempt tosatisfy a desired spatial distribution, such as the Latin square spatialdistribution or the stratified spatial distribution. In an embodiment,step 906 may be modified so as to omit determining the plurality ofrespective sets of pose angle values, or may be modified so thatdetermining the plurality of respective sets of pose angle values isperformed in some other manner that does not involve selecting a surfacepoint from within a surface region on an imaginary sphere. For instance,for such a modified step 906, each of the pose angle values in arespective set of pose angle values may be determined randomly based ona uniform probability distribution function, as discussed above. In thisembodiment, steps 902 and 904 may be omitted, or may still be included,and step 906 may still involve determining a plurality of poses. Theplurality of poses may be determined by determining respective locationsfor the plurality of poses, wherein the respective locations may bedetermined so as to satisfy a desired spatial distribution, such as theLatin square spatial distribution or the stratified spatialdistribution. For instance, such a modified step 906 may involvedetermining a grid that divides a space within a camera field of viewinto one or more layers of multiple rows of 3D regions and multiplecolumns of 3D regions, and determining respective locations forcandidate poses such that the candidate poses will result inrobot-achievable candidate poses which satisfy the Latin square spatialdistribution or the stratified spatial distribution, as discussed above.Such a modified step 906 may further result in a plurality of poses inwhich, for each layer of the grid, each of the rows includes no morethan one pose of the plurality of poses, and each column includes nomore than one pose of the plurality of poses.

Returning to FIG. 9, the method 900 may further include a step 908, inwhich the control circuit 111 outputs a plurality of movement commands(also referred to as robot movement commands) for controlling placementof the calibration pattern. For instance, the robot movement commandsmay include a plurality of motor commands for controlling the robot150/250 to place the calibration pattern 160/260 to a particular pose,which may involve moving the calibration pattern 160/260 to a particularlocation of the pose, and/or tilting the calibration pattern 160/260 toa particular pattern orientation of the pose. In some instances, therobot movement commands may be based on the respective sets of poseangle values determined for the poses that were determined in step 906.In some instances, the robot movement commands may be determined basedon an inverse kinematic function that determines a robot movementcommand based on a desired pose.

In an embodiment, the method 900 may include a step 910, in which thecontrol circuit further receive a plurality of calibration images,wherein each calibration image of the plurality of calibration imagesrepresents (e.g., captures) the calibration pattern and is generatedwhile the calibration pattern has a respective pose of the plurality ofposes. For instance, if eight poses are determined in step 906, then thecontrol circuit 111 in step 910 may receive eight calibration images. Insome cases, the camera 170/270 may have photographed or otherwise imagedthe calibration pattern 160/260 while the calibration pattern 160/260 isat each of the eight poses, so as to generate the eight poses. In someimplementations, the control circuit 111 in step 910 may generate cameracommands which cause the camera 170/270 to photograph the calibrationpattern 160/260, and may output the camera commands (e.g., via thecommunication interface 113) to the camera 170/270. In an embodiment,the control circuit 111 may receive the plurality of calibration imagesfrom the camera 170/270, such as via the communication interface 113. Inan embodiment, the control circuit 111 may receive the plurality ofcalibration images from a storage device on which the calibration imagesare stored, such as the non-transitory computer-readable medium 115, orfrom some other non-transitory computer-readable medium.

In an embodiment, the method 900 may further include a step 912, inwhich the control circuit 111 determines an estimate of a cameracalibration parameter based on the plurality of calibration images. Asstated above, the camera calibration parameter may be an intrinsiccamera calibration parameter, such as a projection matrix or a lensdistortion parameter of the camera 170/270, or may be a parameter whichdescribes a spatial relationship between the camera 170/270 and itsenvironment, such as a location and orientation of the camera 170/270relative to the robot 150/250. In an embodiment, the control circuit 111may determine the estimate of the camera calibration parameter based onequations which describe a relationship between defined locations ofpattern elements (e.g., dots) on the calibration pattern 160/260 in apattern coordinate system and locations at which the pattern elementsappear in the calibration images. Determining an estimate of a cameracalibration parameter is described in more detail in U.S. patentapplication Ser. No. 16/295,940, entitled “METHOD AND SYSTEM FORPERFORMING AUTOMATIC CAMERA CALIBRATION FOR ROBOT CONTROL,” the contentof which is incorporated by reference herein in its entirety.

In an embodiment, the control circuit may be configured, after thecamera calibration is performed, to receive a subsequent image from thecamera via the communication interface, and to output a subsequent robotmovement command that is generated based on the subsequent image andbased on the estimate of the camera calibration parameter. For instance,the subsequent image may be that of a package or stack of packages in awarehouse that are to be de-palletized by the robot 150/250. In someinstances, the control circuit 111 may be configured to determine aspatial relationship between the robot 150/250 and the package, and/or aspatial relationship between the camera 170/270 and the package, basedon the image of the package and based on the estimate of the cameracalibration parameter determined in step 912, as also described in moredetail in U.S. patent application Ser. No. 16/295,940, entitled “METHODAND SYSTEM FOR PERFORMING AUTOMATIC CAMERA CALIBRATION FOR ROBOTCONTROL,” the content of which is incorporated by reference herein inits entirety. The control circuit 111 may then be configured to generatea robot movement command based on the determined spatial relationshipbetween the package and the robot 150/250 or the camera 170/270, andoutput the robot movement command to the robot 150/250.

Concise Description of Various Embodiments

Embodiment 1 relates to a computing system comprising a communicationinterface and a control circuit. The communication interface isconfigured to communicate with a robot and with a camera having a camerafield of view, wherein the robot has a calibration pattern disposedthereon. The control circuit is configured, when the computing system isin communication with the robot and with the camera, to perform cameracalibration by: determining a range of pattern orientations forperforming the camera calibration, wherein the range of patternorientations is a range of orientations for the calibration pattern;determining a surface region on a surface of an imaginary sphere,wherein the surface of the imaginary sphere represents possible patternorientations for the calibration pattern, and the surface regionrepresents the range of pattern orientations for performing the cameracalibration; determining a plurality of poses for the calibrationpattern to adopt when the camera calibration is being performed, whereinthe plurality of poses are defined by respective combinations of aplurality of respective locations within the camera field of view and aplurality of respective sets of pose angle values, wherein each set ofpose angle values of the plurality of respective sets is based on arespective surface point selected from within the surface region on thesurface of the imaginary sphere; outputting a plurality of robotmovement commands for controlling placement of the calibration pattern,wherein the plurality of robot movement commands are generated based onthe plurality of poses that are determined; receiving a plurality ofcalibration images, wherein each calibration image of the plurality ofcalibration images represents the calibration pattern and is generatedwhile the calibration pattern has a respective pose of the plurality ofposes; and determining an estimate of a camera calibration parameterbased on the plurality of calibration images. The control circuit isfurther configured, after the camera calibration is performed, toreceive a subsequent image from the camera via the communicationinterface, and to output a subsequent robot movement command that isgenerated based on the subsequent image and based on the estimate of thecamera calibration parameter.

Embodiment 2 includes the computing system of embodiment 1. In thisembodiment, the control circuit is configured, for respective surfacepoints on which the respective sets of pose angle values are based, torandomly select each of the respective surface points from within thesurface region according to a uniform probability distribution.

Embodiment 3 includes the computing system of embodiment 2. In thisembodiment, the control circuit is configured, for the respectivesurface points on which the respective sets of pose angle values arebased, to randomly select each of the respective surface points fromamong only a uniform set of surface points, wherein the uniform set ofsurface points is a set of surface points that are uniformly distributedwithin the surface region on the surface of the imaginary sphere.

Embodiment 4 includes the computing system of any one of embodiments1-3. In this embodiment, the surface region on the surface of theimaginary sphere forms a circular band of uniform width.

Embodiment 5 includes the computing system of any one of embodiments1-4. In this embodiment, each set of the pose angle values of theplurality of sets of pose angle values is a set of angle values thatrepresent respective amounts of rotation of the calibration patternabout respective axes of rotation, wherein the respective axes areorthogonal to each other, and wherein each of the respective axes isparallel with or orthogonal to a camera optical axis.

Embodiment 6 includes the computing system of embodiment 5. In thisembodiment, each surface point on the surface of the imaginary sphererepresents a respective pattern orientation for the calibration patternthat would cause a normal vector of the calibration pattern to point tothe surface point. Further, the control circuit is configured todetermine each set of pose angle values for the plurality of sets basedon a respective surface point by applying an arctangent function to arespective coordinate for the respective surface point.

Embodiment 7 includes the computing system of any one of embodiments1-6. In this embodiment, the control circuit is configured to determinethe plurality of poses by: determining a grid of 3D regions that dividea space within the camera field of view into one or more layers thateach has multiple rows of 3D regions and multiple columns of 3D regions;determining the plurality of locations for the plurality of poses suchthat the plurality of poses have a spatial distribution within the gridin which, for each layer of the one or more layers: (i) each row of themultiple rows within the layer includes no more than one pose of theplurality of poses and (ii) each column of the multiple columns withinthe layer includes no more than one pose of the plurality of poses.

Embodiment 8 includes the computing system of any one of embodiments1-6. In this embodiment, the control circuit is configured to determinethe plurality of poses by: determining a grid of 3D regions that dividea space within the camera field of view into one or more layers thateach has multiple rows of 3D regions and multiple columns of 3D regions;determining the plurality of locations for the plurality of poses suchthat the plurality of poses have a spatial distribution within the gridin which, for each layer of the one or more layers: (i) each row of themultiple rows within the layer includes no more than one pose of theplurality of poses, or (ii) each column of the multiple columns withinthe layer includes no more than one pose of the plurality of poses.

Embodiment 9 includes the computing system of any one of embodiments1-6. In this embodiment, the control circuit is configured to determinethe plurality of poses by: (a) determining a set of candidate poses,wherein each candidate pose of the set of candidate poses is determinedby: determining a respective location within the camera field of viewfor the candidate pose, selecting a respective surface point from withinthe surface region on the surface of the imaginary sphere, anddetermining a respective set of pose angle values for the candidate posebased on the surface point that is selected; (b) determining a set ofrobot-achievable candidate poses by: determining, for each candidatepose of the set of candidate poses, whether the candidate pose isrobot-achievable, and adding the candidate pose to the set ofrobot-achievable candidate poses in response to a determination that thecandidate pose is robot-achievable; and selecting the plurality of posesfrom among only the set of robot-achievable candidate poses.

Embodiment 10 includes the computing system of embodiment 9. In thisembodiment, the control circuit is configured to determine a grid of 3Dregions that divide a space within the camera field of view into one ormore layers that each has multiple rows of 3D regions and multiplecolumns of 3D regions. Further, the control circuit is configured todetermine a respective location for each candidate pose of the set ofcandidate poses to be a location which is in a layer of the one or morelayers of the grid and which i) does not share a row with anyrobot-achievable candidate pose of the set of robot-achievable candidateposes in that layer, and ii) does not share a column with anyrobot-achievable candidate pose of the set of robot-achievable candidateposes in that layer.

Embodiment 11 includes the computing system of embodiment 9. In thisembodiment, the control circuit is configured to: determine a targetnumber that indicates how many poses are desired for the plurality ofposes; determine a grid size of n based on the target number of poses;determine a grid of 3D regions that divide a space within the camerafield of view into one or more layers that each has n rows of 3D regionsand n columns of 3D regions; determine, for each layer of the one ormore layers and as part of the set of robot-achievable candidate poses,a respective subset of n robot-achievable candidate poses based on aninitial condition that the n robot-achievable candidate poses have nlocations with a first spatial distribution in which i) each row of then rows of the layer includes only one robot-achievable candidate pose,and ii) each column of the n columns of the layer includes only onerobot-achievable candidate pose.

Embodiment 12 includes the computing system of embodiment 11. In thisembodiment, the control circuit is further configured to determine theset of robot-achievable candidate poses by performing the following foreach layer of the one or more layers of the grid: (a) determiningwhether the respective subset of n robot-achievable candidate poses forthe layer are determinable if the respective subset of nrobot-achievable candidate poses have to satisfy the initial condition,wherein the initial condition is a first condition, and (b) in responseto a determination that the respective subset of n robot-achievablecandidate poses are not determinable if the respective subset of nrobot-achievable candidate poses have to satisfy the initial condition,determining the respective subset of n robot-achievable candidate posesbased on a second condition in which the n robot-achievable candidateposes have n locations with a second spatial distribution in which i)each row of the multiple rows of the layer includes only onerobot-achievable candidate pose, or ii) each column of the multiplecolumns of the layer includes only one robot-achievable candidate pose.

Embodiment 13 includes the computing system of embodiment 12. In thisembodiment, the control circuit is further configured to determine theset of robot-achievable candidate poses by further performing thefollowing for each layer of the one or more layers of the grid: (a)determining whether the respective subset of n robot-achievablecandidate poses for the layer are determinable if the respective subsetof n robot-achievable candidate poses have to satisfy the secondcondition, and (b) in response to a determination that the respectivesubset of n robot-achievable candidate poses are not determinable if therespective subset of n robot-achievable candidate poses have to satisfythe second condition, determining the respective subset of nrobot-achievable candidate poses based on a third condition in which then robot-achievable candidate poses have n locations that are randomlydistributed within n respective 3D regions of the layer.

Embodiment 14 includes the computing system of embodiment 12 or 13. Inthis embodiment, the grid has n layers, and wherein the grid size of nis determined by: determining a square root of the target number ofposes for the plurality of poses, and determining the grid size of n asa smallest integer that is greater than or equal to the square root ofthe target number of poses.

Embodiment 15 relates to a computing system comprising a communicationinterface and a control circuit. The communication interface isconfigured to communicate with a robot and with a camera having a camerafield of view, wherein the robot has a calibration pattern disposedthereon. The control circuit is configured, when the computing system isin communication with the robot and with the camera, to perform cameracalibration by: determining a plurality of poses for the calibrationpattern to adopt when the camera calibration is being performed, whereinthe plurality of poses are defined by respective combinations of aplurality of respective locations within the camera field of view and aplurality of pattern orientations; outputting a plurality of robotmovement commands for controlling placement of the calibration pattern,wherein the plurality of robot movement commands are generated based onthe plurality of poses that are determined; receiving a plurality ofcalibration images, wherein each calibration image of the plurality ofcalibration images represents the calibration pattern and is generatedwhile the calibration pattern has a respective pose of the plurality ofposes; and determining an estimate of a camera calibration parameterbased on the plurality of calibration images. The control circuit isfurther configured, after the camera calibration is performed, toreceive a subsequent image from the camera via the communicationinterface, and to output a subsequent robot movement command that isgenerated based on the subsequent image and based on the estimate of thecamera calibration parameter.

Embodiment 16 includes the computing system of embodiment 15. In thisembodiment, the control circuit is configured to determine the pluralityof poses by: (a) determining a grid of 3D regions that divide a spacewithin the camera field of view into one or more layers that each hasmultiple rows of 3D regions and multiple columns of 3D regions; (b)determining the plurality of locations for the plurality of poses suchthat the plurality of poses have a spatial distribution within the gridin which, for each layer of the one or more layers: (i) each row of themultiple rows within the layer includes no more than one pose of theplurality of poses and (ii) each column of the multiple columns withinthe layer includes no more than one pose of the plurality of poses.

Embodiment 17 includes the computing system of embodiment 15. In thisembodiment, the computing system is configured to determine theplurality of poses by: (a) determining a grid of 3D regions that dividea space within the camera field of view into one or more layers thateach has multiple rows of 3D regions and multiple columns of 3D regions;(b) determining the plurality of locations for the plurality of posessuch that the plurality of poses have a spatial distribution within thegrid in which, for each layer of the one or more layers: (i) each row ofthe multiple rows within the layer includes no more than one pose of theplurality of poses, or (ii) each column of the multiple columns withinthe layer includes no more than one pose of the plurality of poses.

Embodiment 18 includes the computing system of embodiment 15. In thisembodiment, the computing system is configured to determine theplurality of poses by: (a) determining a set of candidate poses, whereineach candidate pose of the set of candidate poses is determined by:determining a respective location within the camera field of view forthe candidate pose, (b) determining a set of robot-achievable candidateposes by: determining, for each candidate pose of the set of candidateposes, whether the candidate pose is robot-achievable, and adding thecandidate pose to the set of robot-achievable candidate poses inresponse to a determination that the candidate pose is robot-achievable;and (c) selecting the plurality of poses from among only the set ofrobot-achievable candidate poses.

Embodiment 19 includes the computing system of embodiment 18. In thisembodiment, the control circuit is configured to determine a grid of 3Dregions that divide a space within the camera field of view into one ormore layers that each has multiple rows of 3D regions and multiplecolumns of 3D regions, and wherein the control circuit is configured todetermine a respective location for each candidate pose of the set ofcandidate poses to be a location which is in a layer of the one or morelayers of the grid and which i) does not share a row with anyrobot-achievable candidate pose of the set of robot-achievable candidateposes in that layer, and ii) does not share a column with anyrobot-achievable candidate pose of the set of robot-achievable candidateposes in that layer.

Embodiment 20 includes the computing system of embodiment 18. In thisembodiment, the control circuit is configured to: determine a targetnumber that indicates how many poses are desired for the plurality ofposes; determine a grid size of n based on the target number of poses;determine a grid of 3D regions that divide a space within the camerafield of view into one or more layers that each has n rows of 3D regionsand n columns of 3D regions; determine, for each layer of the one ormore layers and as part of the set of robot-achievable candidate poses,a respective subset of n robot-achievable candidate poses based on aninitial condition that the n robot-achievable candidate poses have nlocations with a first spatial distribution in which i) each row of then rows of the layer includes only one robot-achievable candidate pose,and ii) each column of the n columns of the layer includes only onerobot-achievable candidate pose.

Embodiment 21 includes the computing system of embodiment 20. In thisembodiment, the control circuit is further configured to determine theset of robot-achievable candidate poses by performing the following foreach layer of the one or more layers of the grid: (a) determiningwhether the respective subset of n robot-achievable candidate poses forthe layer are determinable if the respective subset of nrobot-achievable candidate poses have to satisfy the initial condition,wherein the initial condition is a first condition, and (b) in responseto a determination that the respective subset of n robot-achievablecandidate poses are not determinable if the respective subset of nrobot-achievable candidate poses have to satisfy the initial condition,determining the respective subset of n robot-achievable candidate posesbased on a second condition in which the n robot-achievable candidateposes have n locations with a second spatial distribution in which i)each row of the multiple rows of the layer includes only onerobot-achievable candidate pose, or ii) each column of the multiplecolumns of the layer includes only one robot-achievable candidate pose.

Embodiment 22 includes the computing system of embodiment 21. In thisembodiment, the control circuit is further configured to determine theset of robot-achievable candidate poses by further performing thefollowing for each layer of the one or more layers of the grid: (a)determining whether the respective subset of n robot-achievablecandidate poses for the layer are determinable if the respective subsetof n robot-achievable candidate poses have to satisfy the secondcondition, and (b) in response to a determination that the respectivesubset of n robot-achievable candidate poses are not determinable if therespective subset of n robot-achievable candidate poses have to satisfythe second condition, determining the respective subset of nrobot-achievable candidate poses based on a third condition in which then robot-achievable candidate poses have n locations that are randomlydistributed within n respective 3D regions of the layer.

Embodiment 23 includes the computing system of any one of embodiments20-22, wherein the grid has n layers, and wherein the grid size of n isdetermined by: (a) determining a square root of the target number ofposes for the plurality of poses, and (b) determining the grid size of nas a smallest integer that is greater than or equal to the square rootof the target number of poses.

Embodiment 24 includes the computing system of any one of embodiments15-23, wherein the plurality of respective pattern orientations aredefined by a plurality of respective sets of pose angle values, andwherein the control circuit is configured to: determine a range ofpattern orientations for performing the camera calibration, wherein therange of pattern orientations is a range of orientations for thecalibration pattern; determine a surface region on a surface of animaginary sphere, wherein the surface of the imaginary sphere representspossible pattern orientations for the calibration pattern, and thesurface region represents the range of pattern orientations forperforming the camera calibration; determine each set of pose anglevalues of the plurality of respective sets based on a respective surfacepoint selected from within the surface region on the surface of theimaginary sphere (e.g., selected based on a uniform probabilitydistribution). For instance, the above technique for embodiment 24 maybe used in embodiment 18. In such an instance, the control circuit isconfigured to determine the plurality of poses by: (a) determining a setof candidate poses, wherein each candidate pose of the set of candidateposes is determined by: determining a respective location within thecamera field of view for the candidate pose, selecting a respectivesurface point from within the surface region on the surface of theimaginary sphere, and determining a respective set of pose angle valuesfor the candidate pose based on the surface point that is selected; (b)determining a set of robot-achievable candidate poses by: determining,for each candidate pose of the set of candidate poses, whether thecandidate pose is robot-achievable, and adding the candidate pose to theset of robot-achievable candidate poses in response to a determinationthat the candidate pose is robot-achievable; and selecting the pluralityof poses from among only the set of robot-achievable candidate poses.

While various embodiments have been described above, it should beunderstood that they have been presented only as illustrations andexamples of the present invention, and not by way of limitation. It willbe apparent to persons skilled in the relevant art that various changesin form and detail can be made therein without departing from the spiritand scope of the invention. Thus, the breadth and scope of the presentinvention should not be limited by any of the above-described exemplaryembodiments, but should be defined only in accordance with the appendedclaims and their equivalents. It will also be understood that eachfeature of each embodiment discussed herein, and of each reference citedherein, can be used in combination with the features of any otherembodiment. All patents and publications discussed herein areincorporated by reference herein in their entirety.

What is claimed is:
 1. A computing system comprising: a communicationinterface configured to communicate with a robot and with a camerahaving a camera field of view, wherein the robot has a calibrationpattern disposed thereon; and a control circuit configured, when thecomputing system is in communication with the robot and with the camera,to perform camera calibration by: determining a range of patternorientations for performing the camera calibration, wherein the range ofpattern orientations is a range of orientations for the calibrationpattern; determining a surface region on a surface of an imaginarysphere, wherein the surface of the imaginary sphere represents possiblepattern orientations for the calibration pattern, and the surface regionrepresents the range of pattern orientations for performing the cameracalibration; determining a plurality of poses for the calibrationpattern to adopt when the camera calibration is being performed, whereinthe plurality of poses are defined by respective combinations of aplurality of respective locations within the camera field of view and aplurality of respective sets of pose angle values, wherein each set ofpose angle values of the plurality of respective sets is based on arespective surface point selected from within the surface region on thesurface of the imaginary sphere; outputting a plurality of robotmovement commands for controlling placement of the calibration pattern,wherein the plurality of robot movement commands are generated based onthe plurality of poses that are determined; receiving a plurality ofcalibration images, wherein each calibration image of the plurality ofcalibration images represents the calibration pattern and is generatedwhile the calibration pattern has a respective pose of the plurality ofposes; and determining an estimate of a camera calibration parameterbased on the plurality of calibration images, wherein the controlcircuit is further configured, after the camera calibration isperformed, to receive a subsequent image from the camera via thecommunication interface, and to output a subsequent robot movementcommand that is generated based on the subsequent image and based on theestimate of the camera calibration parameter.
 2. The computing system ofclaim 1, wherein the control circuit is configured, for respectivesurface points on which the respective sets of pose angle values arebased, to randomly select each of the respective surface points fromwithin the surface region according to a uniform probabilitydistribution.
 3. The computing system of claim 2, wherein the controlcircuit is configured, for the respective surface points on which therespective sets of pose angle values are based, to randomly select eachof the respective surface points from among only a uniform set ofsurface points, wherein the uniform set of surface points is a set ofsurface points that are uniformly distributed within the surface regionon the surface of the imaginary sphere.
 4. The computing system of claim1, wherein the surface region on the surface of the imaginary sphereforms a circular band of uniform width.
 5. The computing system of claim1, wherein each set of the pose angle values of the plurality of sets ofpose angle values is a set of angle values that represent respectiveamounts of rotation of the calibration pattern about respective axes ofrotation, wherein the respective axes are orthogonal to each other, andwherein each of the respective axes is parallel with or orthogonal to acamera optical axis.
 6. The computing system of claim 5, wherein eachsurface point on the surface of the imaginary sphere represents arespective pattern orientation for the calibration pattern that wouldcause a normal vector of the calibration pattern to point to the surfacepoint, and wherein the control circuit is configured to determine eachset of pose angle values for the plurality of sets based on a respectivesurface point by applying an arctangent function to a respectivecoordinate for the respective surface point.
 7. The computing system ofclaim 1, wherein the control circuit is configured to determine theplurality of poses by: determining a grid of 3D regions that divide aspace within the camera field of view into one or more layers that eachhas multiple rows of 3D regions and multiple columns of 3D regions;determining the plurality of locations for the plurality of poses suchthat the plurality of poses have a spatial distribution within the gridin which, for each layer of the one or more layers: (i) each row of themultiple rows within the layer includes no more than one pose of theplurality of poses and (ii) each column of the multiple columns withinthe layer includes no more than one pose of the plurality of poses. 8.The computing system of claim 1, wherein the control circuit isconfigured to determine the plurality of poses by: determining a grid of3D regions that divide a space within the camera field of view into oneor more layers that each has multiple rows of 3D regions and multiplecolumns of 3D regions; determining the plurality of locations for theplurality of poses such that the plurality of poses have a spatialdistribution within the grid in which, for each layer of the one or morelayers: (i) each row of the multiple rows within the layer includes nomore than one pose of the plurality of poses, or (ii) each column of themultiple columns within the layer includes no more than one pose of theplurality of poses.
 9. The computing system of claim 1, wherein thecontrol circuit is configured to determine the plurality of poses by:determining a set of candidate poses, wherein each candidate pose of theset of candidate poses is determined by: determining a respectivelocation within the camera field of view for the candidate pose,selecting a respective surface point from within the surface region onthe surface of the imaginary sphere, and determining a respective set ofpose angle values for the candidate pose based on the surface point thatis selected; determining a set of robot-achievable candidate poses by:determining, for each candidate pose of the set of candidate poses,whether the candidate pose is robot-achievable, and adding the candidatepose to the set of robot-achievable candidate poses in response to adetermination that the candidate pose is robot-achievable; and selectingthe plurality of poses from among only the set of robot-achievablecandidate poses.
 10. The computing system of claim 9, wherein thecontrol circuit is configured to determine a grid of 3D regions thatdivide a space within the camera field of view into one or more layersthat each has multiple rows of 3D regions and multiple columns of 3Dregions, and wherein the control circuit is configured to determine arespective location for each candidate pose of the set of candidateposes to be a location which is in a layer of the one or more layers ofthe grid and which i) does not share a row with any robot-achievablecandidate pose of the set of robot-achievable candidate poses in thatlayer, and ii) does not share a column with any robot-achievablecandidate pose of the set of robot-achievable candidate poses in thatlayer.
 11. The computing system of claim 9, wherein the control circuitis configured to: determine a target number that indicates how manyposes are desired for the plurality of poses; determine a grid size of nbased on the target number of poses; determine a grid of 3D regions thatdivide a space within the camera field of view into one or more layersthat each has n rows of 3D regions and n columns of 3D regions;determine, for each layer of the one or more layers and as part of theset of robot-achievable candidate poses, a respective subset of nrobot-achievable candidate poses based on an initial condition that then robot-achievable candidate poses have n locations with a first spatialdistribution in which i) each row of the n rows of the layer includesonly one robot-achievable candidate pose, and ii) each column of the ncolumns of the layer includes only one robot-achievable candidate pose.12. The computing system of claim 11, wherein the control circuit isfurther configured to determine the set of robot-achievable candidateposes by performing the following for each layer of the one or morelayers of the grid: determining whether the respective subset of nrobot-achievable candidate poses for the layer are determinable if therespective subset of n robot-achievable candidate poses have to satisfythe initial condition, wherein the initial condition is a firstcondition, and in response to a determination that the respective subsetof n robot-achievable candidate poses are not determinable if therespective subset of n robot-achievable candidate poses have to satisfythe initial condition, determining the respective subset of nrobot-achievable candidate poses based on a second condition in whichthe n robot-achievable candidate poses have n locations with a secondspatial distribution in which i) each row of the multiple rows of thelayer includes only one robot-achievable candidate pose, or ii) eachcolumn of the multiple columns of the layer includes only onerobot-achievable candidate pose.
 13. The computing system of claim 12,wherein the control circuit is further configured to determine the setof robot-achievable candidate poses by further performing the followingfor each layer of the one or more layers of the grid: determiningwhether the respective subset of n robot-achievable candidate poses forthe layer are determinable if the respective subset of nrobot-achievable candidate poses have to satisfy the second condition,and in response to a determination that the respective subset of nrobot-achievable candidate poses are not determinable if the respectivesubset of n robot-achievable candidate poses have to satisfy the secondcondition, determining the respective subset of n robot-achievablecandidate poses based on a third condition in which the nrobot-achievable candidate poses have n locations that are randomlydistributed within n respective 3D regions of the layer.
 14. Thecomputing system of claim 12, wherein the grid has n layers, and whereinthe grid size of n is determined by: determining a square root of thetarget number of poses for the plurality of poses, and determining thegrid size of n as a smallest integer that is greater than or equal tothe square root of the target number of poses.
 15. A non-transitorycomputer-readable medium having instructions stored thereon that, whenexecuted by a control circuit of a computing system, causes the controlcircuit to perform camera calibration when the computing system is incommunication with a camera having a camera field of view and with arobot having a calibration pattern disposed thereon, by: determining arange of pattern orientations for performing the camera calibration,wherein the computing system includes a communication interfaceconfigured to communicate with the robot and with the camera, whereinthe range of pattern orientations is a range of orientations for thecalibration pattern; determining a surface region on a surface of animaginary sphere, wherein the surface of the imaginary sphere representspossible pattern orientations for the calibration pattern, and theregion on the surface represents the range of pattern orientations forperforming the camera calibration; determining a plurality of poses forthe calibration pattern to adopt when the camera calibration is beingperformed, wherein the plurality of poses are defined by respectivecombinations of a plurality of respective locations within the camerafield of view and a plurality of respective sets of pose angle values,wherein each set of pose angle values of the plurality of respectivesets is based on a respective surface point selected from within thesurface region on the surface of the imaginary sphere; outputting aplurality of robot movement commands for controlling placement of thecalibration pattern, wherein the plurality of robot movement commandsare generated based on the plurality of poses that are determined;receiving a plurality of calibration images, wherein each calibrationimage of the plurality of calibration images represents the calibrationpattern and is generated while the calibration pattern has a respectivepose of the plurality of poses; and determining an estimate of a cameracalibration parameter based on the plurality of calibration images,wherein the instructions, when executed by the control circuit after thecamera calibration is performed, further cause the control circuit toreceive a subsequent image from the camera via the communicationinterface, and to output a subsequent robot movement command that isgenerated based on the subsequent image and based on the estimate of thecamera calibration parameter.
 16. The non-transitory computer-readablemedium of claim 15, wherein the instructions, when executed by thecontrol circuit, further cause the control circuit to perform thedetermining of the plurality of poses by: randomly selecting surfacepoints from among only a uniform set of surface points, wherein theuniform set of surface points is a set of surface points that areuniformly distributed within the surface region on the surface of theimaginary sphere, wherein the respective sets of pose angle values arebased on the set of surface points that are selected.
 17. Thenon-transitory computer-readable medium of claim 15, wherein theinstructions, when executed by the control circuit and when thecomputing system is in communication with the robot and the camera,further cause the control circuit to determine the plurality of posesby: determining a grid of 3D regions that divide a space within thecamera field of view into one or more layers that each has multiple rowsof 3D regions and multiple columns of 3D regions; determining theplurality of locations for the plurality of poses such that theplurality of poses have a spatial distribution within the grid in which,for each layer of the one or more layers: (i) each row of the multiplerows within the layer includes no more than one pose of the plurality ofposes and (ii) each column of the multiple columns within the layerincludes no more than one pose of the plurality of poses.
 18. Thenon-transitory computer-readable medium of claim 15, wherein theinstructions, when executed by the control circuit and when thecomputing system is in communication with the robot and the camera,further cause the control circuit to determine the plurality of posesby: determining a set of candidate poses, wherein each candidate pose ofthe set of candidate poses is determined by: determining a respectivelocation within the camera field of view for the candidate pose,selecting a respective surface point from within the surface region onthe surface of the imaginary sphere, and determining a respective set ofpose angle values for the candidate pose based on the surface point thatis selected; determining a set of robot-achievable candidate poses by:determining, for each candidate pose of the set of candidate poses,whether the candidate pose is robot-achievable, and adding the candidatepose to the set of robot-achievable candidate poses in response to adetermination that the candidate pose is robot-achievable; and selectingthe plurality of poses from among only the set of robot-achievablecandidate poses.
 19. A method for performing robot control, wherein themethod comprises: determining, by a computing system, a range of patternorientations for performing camera calibration, wherein the computingsystem includes a communication interface configured to communicate witha robot having a calibration pattern disposed thereon and with a camera;determining, by the computing system, a surface region on a surface ofan imaginary sphere, wherein the surface of the imaginary sphererepresents possible pattern orientations for the calibration pattern,and the region on the surface represents the range of patternorientations for performing the camera calibration; determining, by thecomputing system, a plurality of poses for the calibration pattern toadopt when the camera calibration is being performed, wherein theplurality of poses are defined by respective combinations of a pluralityof respective locations within the camera field of view and a pluralityof respective sets of pose angle values, wherein each set of pose anglevalues of the plurality of respective sets is based on a respectivesurface point selected from within the surface region on the surface ofthe imaginary sphere; outputting, by the computing system, a pluralityof robot movement commands for controlling placement of the calibrationpattern, wherein the plurality of robot movement commands are generatedbased on the plurality of poses that are determined; receiving, by thecomputing system, a plurality of calibration images, wherein eachcalibration image of the plurality of calibration images represents thecalibration pattern and is generated while the calibration pattern has arespective pose of the plurality of poses; determining, by the computingsystem, an estimate of a camera calibration parameter based on theplurality of calibration images; receiving, by the computing systemafter the estimate of the camera calibration parameter is determined, asubsequent image via the communication interface; and outputting, by thecomputing system, a subsequent robot movement command that is generatedbased on the subsequent image and based on the estimate of the cameracalibration parameter.
 20. The method of claim 19, wherein determiningthe plurality of poses comprises randomly selecting surface points fromamong only a uniform set of surface points, wherein the uniform set ofsurface points is a set of surface points that are uniformly distributedwithin the surface region on the surface of the imaginary sphere,wherein the respective sets of pose angle values are based on the set ofsurface points that are selected.